An LLM evaluation framework is a fixed set of test cases, written pass criteria and a grading method that you run on every change to an AI feature. It turns "this feels better" into a number you can compare. Build it before launch, keep it small enough to run often, and grow it from real failures.
This guide is for CTOs, founders and product leads who are about to ship a generative AI feature. It draws on documentation from OpenAI and Anthropic and on one widely cited research paper, and it ends with an order of work.
Why "it looks good" is not a test
OpenAI's evaluation best practices name the most common failure. They call it vibe-based evaluation: judging by impressions, or skipping evaluation until after the product ships. The same page asks teams to evaluate early and often, and to treat evaluation as a continuous process.
The reason is simple. Generative output varies. A prompt tweak that fixes one case can break three others, and you will not see it by reading a few answers. A fixed test set shows the damage in minutes.
A score from before you change anything. Every later change is judged against it.
The same test set run after each prompt, model or data change.
A written rule such as "no drop on the core set" that decides what ships.
Real bad answers from production, added to the set so they never return.
Step 1: write success criteria first
Anthropic's guide on defining success criteria says good criteria are specific, measurable, achievable and relevant. "Accurate sentiment classification" beats "good performance". The page also notes that most use cases need several criteria at once, such as accuracy, safety, error severity and response time.
Relevance matters too. The same guide points out that citation accuracy matters more for a medical app than for a casual chatbot. So the criteria that count depend on the harm a wrong answer can cause.
A useful criterion has a number and a test set attached. Anthropic's own example is an F1 score of at least 0.85 on a held-out set of 10,000 diverse posts. Your numbers will differ. The shape is what to copy: a metric, a threshold and a defined set.
Step 2: build a test set that matches real use
OpenAI warns about dataset bias, and says test data should faithfully reproduce production traffic patterns. Anthropic gives the same advice: mirror the real task and include edge cases. Its list covers irrelevant or missing input, overly long input, harmful input and ambiguous cases where humans also disagree.
Good places to find cases:
- Real user questions from logs, once you have a pilot.
- Support tickets and search queries your team already holds.
- Known past failures, each added the day it is found.
- Synthetic cases, written by a model from a seed set, then reviewed by a person.
- Adversarial inputs that try to break the feature.
Anthropic also suggests using a model to generate more cases from a baseline of examples, since writing hundreds by hand is hard. Have a person check them. Weak cases give false confidence.
- 1Define the jobState what the feature must do and what a wrong answer costs.
- 2Write criteriaPick metrics and thresholds that a reviewer could check.
- 3Collect casesTypical, edge and adversarial inputs, drawn from real traffic where possible.
- 4Choose gradersCode first, then an LLM judge, then humans for calibration.
- 5Run and compareScore a baseline, change one thing, score again.
- 6Gate the releaseShip only when the written rule is met.
Step 3: choose the right grader
Anthropic compares three grading methods. The table below summarizes the tradeoffs it describes.
| Method | Strength | Weakness |
|---|---|---|
| Code-based (exact or string match) | Fastest, most reliable, scales well | Misses nuance in complex answers |
| LLM-based | Fast, flexible, handles complex judgments | Must be checked for reliability before scaling |
| Human | Most flexible, highest quality | Slow and expensive |
Anthropic's rule is to pick the fastest, most reliable and most scalable method that works. That usually means code checks for anything with a clear right answer, such as a label, a field or a required phrase. OpenAI adds that metric scores such as exact match are good for regression testing but may miss nuance, so read them next to human judgment.
Anthropic also says that many automatically graded cases are better than a few hand-graded ones. That is a call about signal per hour of effort. It does not mean humans have no role, as the next section shows.
Step 4: use LLM judges with care
An LLM judge reads an answer and scores it against a rubric. It lets you grade tone, helpfulness or faithfulness at scale. The research behind this is the paper Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Its abstract reports that strong judges reach over 80% agreement with human preferences, the same level humans reach with each other. It also names position bias, verbosity bias and self-enhancement bias, and limited reasoning ability.
OpenAI lists the same traps. Judges can favor the answer shown first, or the longer answer. Its suggested fixes include pairwise or pass/fail grading, controlling for response length, and asking for reasoning before the score.
Write the rubric in plain, checkable terms. Anthropic's example is a rule that the answer must mention a named product in its first sentence, or it is graded incorrect. Ask for "correct" or "incorrect", or a 1 to 5 scale, not an open opinion.
Common mistakes to avoid
- Testing on the cases you tuned on. Keep a held-out set that you never use while editing prompts. Anthropic's development cycle ends with a check against held-out evals.
- Only generic metrics. OpenAI lists overly generic metrics, such as relying only on perplexity or BLEU, as an anti-pattern.
- Never calibrating. OpenAI lists ignoring human feedback as a pitfall. Check automated scores against human labels on a schedule.
- One big score. A single average hides a failing slice. Report results per case type, and track safety and latency next to quality.
- Letting the set go stale. Production changes. Add new failures every week.
A practical order of work
Start with one feature and one page of criteria. Collect a first set of cases you can run in a few minutes. Score a baseline, then change one thing at a time. Add LLM judges only where code checks cannot do the job, and validate each one against human labels.
Teams that want help setting this up can look at our AI development practice, which covers generative AI apps and evaluation harnesses. Test strategy and release sign-off sit with our QA and testing practice. Our process page explains that QA is a gate, not a phase.
If you have a feature in flight and no way to measure it, book a technical call. It is a technical conversation, not a sales pitch, and we can sign an NDA before you share details.
Frequently asked questions
What is an LLM evaluation framework?
It is a repeatable way to measure how well your AI feature does its job. It has a set of test cases, written success criteria, a grading method and a rule for when a change is good enough to ship.
How many test cases do we need to start?
Start small and grow. Anthropic's example table uses test sets from 50 to 1,000 items depending on the task. Begin with a set you can run on every change, then add cases from real failures.
Can an LLM grade another LLM's answers?
Yes, with care. Research on LLM judges reports over 80% agreement with human preferences, but also position, verbosity and self-enhancement biases. Check the judge against human labels before you trust it.
When should humans grade the answers?
Use humans to set the standard and to check the automated graders. Both OpenAI and Anthropic describe human grading as the highest quality but slow and expensive, so use it on a sample, not on every run.
Do we need an evaluation framework before launch?
Yes. OpenAI lists vibe-based evaluation, or skipping evaluation until after shipping, as an anti-pattern. A small set built before launch gives you a baseline to compare every later change against.
Need help with AI and machine learning?
Talk to the engineers who would do the work. We reply within one working day.

