The first time a team ships something built on a language model, the test suite stops meaning what it used to. The same input produces a different answer, so the assertion fails, so someone deletes the assertion. Six months later the only thing standing between a prompt change and production is whether anyone happened to try it.
The problem is real but narrower than it looks, and it has a settled set of answers.
Start by shrinking the problem
A feature built on a model is mostly ordinary software. Retrieval, prompt assembly, output parsing, permission checks, rate limiting, error handling, storage: all deterministic, all testable exactly as before. The non-deterministic part is one function call in the middle.
Draw that boundary explicitly and put a seam at it. With the model call behind an interface you can substitute, the surrounding code gets a normal test suite with recorded responses, including the ugly ones: truncated output, malformed JSON, a refusal, a timeout, a response in the wrong language. Those are the failures that actually take features down, and every one of them is deterministic to test.
A useful consequence: once the shell is covered by fast deterministic tests, the expensive evaluation runs only need to answer one question, which is whether the model’s output is good. That is a much smaller question than “does the feature work”, and it is the only part that needs the slow, costly machinery described below.
Four scorers, cheapest first
Assertions. Deterministic checks on the output, regardless of its wording. Does it parse as JSON and validate against the schema. Are the required fields present and non-empty. Does every cited document ID exist in the corpus that was actually retrieved. Is the number inside a plausible range. Is the answer under the length limit. Does it avoid the strings you never want to see.
That last family catches more than its reputation suggests. Hallucinated citations, the failure people worry about most in retrieval systems, are frequently catchable with an exact-match check against retrieved document IDs. No judgement required, and it costs nothing.
A golden set. A curated collection of inputs with expected outcomes: common cases, edge cases and every failure you have already seen in production. Two hundred examples is a commonly cited working size, and it is enough. The value is not in the number, it is in the fact that each one was added for a reason.
A judge model. For qualities that have no exact answer: is the tone right, is the answer complete, is it faithful to the passages it was given. Genuinely useful and genuinely fragile, which the next section is about.
Human review. The only method that finds problems nobody thought to define. Expensive, so sample it rather than scaling it, and route the samples from live traffic rather than from your test set.
Using a judge model without fooling yourself
Asking a model to grade another model’s output works, up to a point, and the point is worth knowing precisely.
An empirical study of the method by Yamauchi, Yano and Oyamada, published in June 2025, examined which design choices affect how closely a judge agrees with humans. Three findings are directly actionable.
Written criteria matter more than anything else
Explicit evaluation criteria were the single most important factor in reliability. A judge prompt saying “rate the quality from 1 to 5” produces numbers that look like data and are not. Write down what a 4 means and what a 2 means, in the same language you would use to brief a human reviewer.
Chain-of-thought adds little once criteria are clear
Asking the judge to reason step by step gave minimal gains when the criteria were well defined. It does cost output tokens, which on most models are priced at five times input. Spend the effort on the rubric rather than the reasoning scaffold, and keep the reasoning only where it helps a human audit the score.
Do not run the judge once and believe it
Non-deterministic sampling aligned better with human preferences than deterministic decoding, and averaging several runs correlates more closely with human judgement than any single run. Score each example three to five times and take the mean or the median. It costs proportionally more and it is the difference between a metric and a coin flip.
Judges are also sensitive to things that should not matter. Paraphrasing, formatting and the order in which options are presented can all change the verdict. If you are comparing two candidate outputs, present each pair in both orders and average, or you are measuring position preference rather than quality. And never use the same model, with the same prompt, as both author and judge without checking that against human scores first.
Making it fail the build without making it annoying
The mechanics in continuous integration are straightforward. Run the golden set on every change that touches a prompt, a model version, a retrieval parameter or the assembly logic. Compare each metric to a stored baseline. Fail the job on a regression.
The part people get wrong is the threshold.
Three rules make this survive contact with a real team.
- Average across several runs before comparing. A single run against a baseline is comparing one sample to another sample. Three to five, meaned, is the minimum that means anything.
- Set the band from measured variance. Run the unchanged system twenty times, look at the spread, and put the threshold outside it. A band chosen by intuition either never fires or fires constantly.
- Treat a persistent drop differently from a single one. One run outside the band is worth a comment on the pull request. Three consecutive runs outside it is a build failure. This distinction is what stops the suite from being disabled.
A suite that cries wolf is worse than no suite, because it teaches the team to override it, and the override becomes reflex long before the real regression arrives.
Two things that go stale
Your golden set. It represents the failures you knew about when you wrote it. Production generates new ones continuously. Sample live traffic, have a human look at a slice of it weekly, and promote every genuine failure into the set. A golden set that has not grown in three months is measuring a system that no longer exists.
The model underneath you. Providers update and deprecate models. The version you evaluated is not guaranteed to be the version serving traffic unless you pinned it, and even a pinned version has an end date. Pin explicitly, record the version alongside every evaluation result, and run the full set before any model change rather than after the complaints.
Alongside the offline suite, keep a small number of online checks running against live traffic: the rate of schema failures, the rate of empty or refused answers, the rate of citations that do not resolve. These are cheap assertions on real inputs and they catch the failures your test set was never going to contain.
What a workable setup looks like
None of this is exotic. It is the same discipline as any other test suite, applied to a component whose output is a distribution rather than a value. The teams that struggle are usually the ones that concluded early on that the system could not be tested, and then built everything else on that assumption.
More on what all this machinery costs to run in what it costs to run an LLM feature in production, on the architecture choices that change what you need to evaluate in RAG, fine-tuning or prompting, and on the retrieval permissions that no quality metric will catch for you in building an AI feature on your own data without leaking it.
Our AI and machine learning team builds the evaluation harness alongside the feature, because retrofitting one onto a system already in production means starting without any of the failures that would have populated it. If you have something working in a demo and no way to tell whether a change made it worse, tell us what it does.
Common questions
How do you test software that gives a different answer every time?
Put a seam at the model call so the surrounding deterministic code is tested normally with recorded responses, then score the model output with the cheapest method that catches the failure you care about: deterministic assertions on structure and citations, a curated golden set run on every change, a judge model for open-ended quality, and a human sample for what nobody defined. Compare against a baseline using a tolerance band rather than an exact value.
What is a golden dataset for LLM evaluation?
A curated set of inputs paired with expected outcomes, covering common cases, edge cases and every failure already seen in production. Around two hundred examples is a commonly used working size. Its value comes from each entry having been added for a specific reason, and from continuing to grow as production surfaces new failures. A set that has not grown in three months is measuring a system that no longer exists.
Is LLM-as-a-judge reliable?
Reliable enough to be useful, with conditions. Empirical work published in June 2025 found that explicit written evaluation criteria are the most important factor, that chain-of-thought reasoning adds little once criteria are clear, and that averaging several non-deterministic runs correlates better with human judgement than a single deterministic one. Judges are also sensitive to paraphrasing, formatting and the order in which options appear, so present comparisons in both orders and average.
How do you stop AI evaluation tests failing randomly in CI?
Average each metric over three to five runs before comparing, set the tolerance band from the measured run-to-run variance of the unchanged system rather than from intuition, and fail the build only on a persistent drop across several consecutive runs rather than a single excursion. A suite that fires on normal variance gets overridden within a fortnight, and the override becomes reflex before the real regression arrives.
Sources
- Yusuke Yamauchi, Taro Yano and Masafumi Oyamada, An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability, arXiv:2506.13639, submitted 16 June 2025. Evaluated against the BIGGENBench and EvalBiasBench benchmarks on open-ended instruction-following tasks. Found that explicit evaluation criteria are critical to reliability, that non-deterministic sampling aligns better with human preferences than deterministic decoding, and that chain-of-thought reasoning yields minimal gains once clear criteria are present.
- The finding that averaging scores across runs, by mean, median or majority, correlates more closely with human judges than single deterministic decoding is reported in the same body of work on judge design, and is consistent with the sensitivity of judges to paraphrasing, formatting and option ordering documented across the LLM-as-judge literature.
- The golden-set working size of roughly two hundred representative examples is a figure in common practice rather than a measured optimum, and should be treated as a starting point rather than a target.
- The charts above are illustrations of the pattern being described, explicitly labelled as such, not measured data from a specific system.
Written September 2026.
