Eval Harnesses for Non-Deterministic Systems

You have decided to build. The system calls a model, the model calls tools, and the same input produces a different trace on two consecutive runs. The question is no longer whether that is acceptable — you own the outcome — but how you will know, before a release goes out, whether the thing got better or worse.

A test suite that asserts equality against a golden string will go red on the first harmless rewording and amber forever after. What replaces it is not a looser version of the same idea. It is a different artefact with a different shape: a harness that treats the system’s output as a distribution, splits decidable properties from judgements, and reports failure modes rather than a score.

What is the harness actually measuring?

Not “is the model good”. The model is a dependency you did not write. What you own is the composition: the prompt assembly, the retrieval policy, the tool surface, the schema, the retry and fallback logic, the context budget. The harness measures that composition against an input distribution you have chosen to serve.

This framing settles several arguments early. A case that fails because the corpus does not contain the answer is not a model failure; it is a coverage failure, and it belongs in a different bucket from a case where the answer was retrieved and then misread. A case that fails because the agent called a tool it should not have access to is not a quality problem at all — it is a boundary defect, and it should be a hard failure regardless of how good the prose was.

So the first design decision is taxonomy. Before you write a single case, enumerate the ways the system can be wrong: schema violation, unauthorised tool call, ungrounded claim, correct-but-unusable formatting, refusal on a valid request, loop without termination, missing citation, stale answer. Each mode gets a name, and every failing case is attributed to exactly one. A harness that cannot tell you which mode regressed is a smoke alarm without a location.

Why does a single score hide the thing you need?

Because the modes trade against each other. Tighten the grounding instruction and refusals rise. Loosen it and ungrounded claims rise. A composite number can sit still through both movements while the behaviour your users actually complain about doubles.

Report per-mode counts over the fixture set, and report them with the spread. Run each case more than once. The same input, the same prompt, the same corpus — and record how often it passes, not whether it passed. A case that passes on some runs and fails on others is the most useful signal the harness produces: it marks a boundary where the system is marginal, and it is exactly the case that will be flaky in production and irreproducible in a bug report.

What belongs in a hard assert?

More than people expect. Non-determinism in the prose does not make every property undecidable. Structured outputs constrain a model’s response to a supplied schema, and schema validity is a boolean you can test without a judge. So is: every citation identifier resolves to a document actually in the corpus; every tool invocation names a tool on the allow-list; every argument type-checks; the trace terminated inside the step budget; no field contains a placeholder token.

These are the cheap checks, and they should run on every case, every time, as a gate. They catch the defects that are genuinely binary and they catch them without rubric drift. Push as much of the surface into this layer as the design allows — that is an argument for schema-constrained generation over free text wherever the downstream consumer is code rather than a person.

How do you build the fixture set without poisoning it?

Draw cases from real traffic, freeze them, and split them by purpose. A regression set exists to prove that behaviour you have already fixed stays fixed; it grows one case at a time, each one attached to a defect, and it never changes except to add. An exploratory set exists to find modes you have not named yet; it rotates, and it is allowed to be messy.

The poisoning risk is in the middle. Once a case is in the regression set and engineers are staring at it daily, the prompt starts to accrete clauses that address that specific case. The harness goes green and the input distribution it represents does not move. Guard against this by keeping a held-back slice that nobody tunes against and that only runs before a release, and by checking that regression cases still resemble current traffic rather than the traffic of whenever they were captured.

Where does a model judge earn its place?

Where the property is real but not decidable in code: is this answer grounded in the retrieved passages, is the tone appropriate for the channel, does the response actually address the question asked. A judge is useful there and nowhere else.

Treat the judge as a component with its own eval. It is non-deterministic in the same way the system under test is, so it needs the same discipline: a written rubric, binary or small-ordinal verdicts rather than a ten-point scale, one property per call, and calibration against human labels on a held set. If the judge and a human disagree on a slice of cases, that disagreement is a defect in the harness, and fixing the rubric comes before fixing the system. A judge that has never been calibrated is producing a number with no referent.

Keep the judge’s inputs tight. Give it the output and the evidence it is meant to check against, not the whole trace, not the prompt, and never the previous verdict. Judges that can see which variant produced the answer will reward the variant.

type Case = {
id: string;
input: string;
corpusScope: string[];
runs: number;
};

const assertions = [
schemaValid("AnswerV2"),
citationsResolve((c) => corpus.has(c.docId)),
toolsWithinAllowList(["search", "fetchPolicy"]),
stepsUnder(8),
];

const graded = [
rubric("grounded", "Every factual claim is supported by a cited passage."),
rubric("responsive", "The answer addresses the question that was asked."),
];

async function evaluate(c: Case) {
const traces = await Promise.all(
  Array.from({ length: c.runs }, () => system.run(c.input, c.corpusScope)),
);
return traces.map((t) => ({
  hard: assertions.map((a) => a(t)),
  soft: graded.map((g) => judge(g, t.output, t.evidence)),
  mode: classifyFailure(t),
}));
}

How do you split retrieval failures from generation failures?

By evaluating them separately, with separate fixtures. Retrieval-augmented generation grounds a model’s output in supplied documents, which means a wrong answer has two possible origins and the harness must distinguish them or you will tune the wrong half of the system.

For retrieval, the case is a query and a set of document identifiers that a human has confirmed contain the answer. The check is whether those identifiers appear in the retrieved set, and at what rank. No judge required. For generation, the case supplies the passages directly — bypassing retrieval entirely — and asks whether the answer follows from them. If retrieval is clean and generation is dirty, the fix is in the prompt or the schema. If retrieval is dirty, no amount of prompt work will help, and a harness that only measures end-to-end will keep sending you to the wrong place.

What does the harness cost to run, and when do you run it?

Every case is model calls, and a harness that runs the full suite on every commit will either be throttled or quietly disabled. Tier it. A smoke tier of hard asserts over a small slice runs on every change and is fast enough that nobody routes around it. The full suite with graded checks runs on merge to the release branch. The held-back slice and the multi-run stability pass run before a release goes out.

Budget the harness explicitly rather than letting it grow until someone kills it. Cases earn their place: a regression case that has not failed since the defect that created it, and whose mode is now covered by a hard assert, can be retired. A suite that only grows becomes a suite that nobody runs.

What does a failing eval actually block?

Decide this before the first red run, because deciding it afterwards is negotiation. A hard assert failure blocks the release — no exceptions, because the property was chosen precisely for being binary. A graded threshold breach does not block; it requires a named person to look at the failing cases and sign off or stop. A new failure mode that the taxonomy does not cover blocks nothing and triggers a taxonomy change.

The discipline is that the harness has authority over the release. If a red run can be waved through by whoever is shipping, the harness is documentation, and documentation does not catch regressions. The value is not in the scores. It is in having an artefact that can say no.