Teams ship LLM features with a test suite that covers the code around the model and nothing that covers the model's behavior itself. Then a prompt tweak, a model upgrade, or a retrieval change silently regresses quality, and the first signal anyone gets is a spike in support tickets or a founder's Slack message. An eval suite is the thing standing between you and that outcome, and it's cheaper to build than most teams assume.
Start with a dataset, not a metric
The eval suite is only as good as the inputs it tests against. Pull real inputs from production logs — actual user queries, actual documents, actual edge cases — rather than inventing synthetic ones, because synthetic test cases tend to be easier than reality and give false confidence. Aim for 50-200 examples that cover: common cases, known hard cases from past incidents, and adversarial or malformed inputs. Label each with what a correct or acceptable response looks like, not necessarily an exact string.
eval_cases = [
{
"input": "cancel my subscription and refund last month",
"checks": ["mentions cancellation confirmation", "does not promise refund without eligibility check"],
},
{
"input": "", # empty input edge case
"checks": ["returns graceful clarification request", "does not throw or hallucinate"],
},
]Pick the right grading method per case, not one for the whole suite
- Exact match / regex for structured outputs (does it return valid JSON, does it contain a required field). Cheap and deterministic — use it whenever the output has a checkable shape.
- Rule-based checks for known constraints (no PII in output, cites a real document ID, response under N tokens).
- LLM-as-judge for open-ended quality (is this helpful, is this grounded in the provided context) — necessary for judging free-text quality but needs its own validation before you trust it.
- Human review as a periodic calibration check on a sample, especially to validate that the LLM judge agrees with human judgment.
Reach for LLM-as-judge last, not first — it's the most expensive and least deterministic option, and a huge fraction of what teams use it for could be a regex or a JSON schema check instead.
LLM-as-judge needs its own eval
An LLM judge is itself a prompt, and it inherits every failure mode of prompting: it can be inconsistent, biased toward longer or more confident-sounding answers, and sensitive to how the grading rubric is worded. Before trusting a judge's scores, validate it against a small human-labeled set — if the judge disagrees with human graders more than a small percentage of the time, fix the rubric before trusting the automated signal. Re-validate whenever you change the judge model, the same way you'd re-validate a fine-tune.
Wire it into the deploy path, not just the dashboard
An eval suite that runs manually before a release gets skipped under deadline pressure. Run it automatically on every prompt, retrieval config, or model change, and block or flag deploys that regress below a threshold on any category of the eval set — not just the aggregate score, since an aggregate can hide a category that dropped to zero while everything else improved slightly.
| Signal | Good for catching |
|---|---|
| Aggregate pass rate | Broad regressions |
| Per-category breakdown | A specific failure mode hidden by averages |
| Latency/cost per eval run | Silent performance regressions alongside quality ones |
Offline evals won't catch everything — production traffic always finds cases your dataset didn't anticipate — so pair the suite with ongoing production monitoring and a habit of feeding real failures back into the eval set. The suite should grow every time something breaks in the wild; that's what keeps it relevant instead of static.
A worked failure mode
A team ships a summarizer with a 50-example golden set scored by an LLM-as-judge. Offline scores stay flat while production tickets spike: the judge and the product model share the same verbosity bias, so long, confident summaries score well even when they drop a negation. A week later a prompt tweak that shortens answers tanks the judge score and gets reverted, even though human raters preferred it. The failure is an eval suite that cannot see the actual error class. The fix is a small, labeled slice for that class (negation, citation hallucination, PII leak), a cheap deterministic check where possible (schema, regex, grounded-span overlap), and a frozen judge prompt versioned like any other dependency. Human review on a rotating sample is the calibration, not a one-time launch ritual.
When this is the wrong tool
If the task is classification with a closed label set, train or prompt a classifier and measure precision/recall; a chat eval harness is overhead. If you cannot name the failure you care about, more eval prompts will not help. Do not use an LLM judge as the only gate for safety or billing-critical output. Offline suites will not catch traffic-shape bugs (timeouts, truncated context, tool schema drift). When the product is a thin wrapper over a model API with no retrieval or tools, a handful of regression prompts may be enough. Build a heavy eval platform only when changes ship weekly and the cost of a silent quality drop exceeds the cost of labeling.