Auditing evaluation score variance before blocking a rollout

A single evaluation score looks precise but often is not. Sampling temperature, nondeterministic GPU kernels, tie breaking in ranking scorers, and judge model variance can move a score by a point or more between identical runs. When a release gate compares one new score against one old score, normal run to run movement can look like a regression.

The operational risk runs in both directions. A noisy drop can block a good rollout, which burns engineering time and teaches teams to ignore the gate. A noisy increase can hide a real regression, which lets a weaker model ship. Both failures come from treating a point estimate as exact.

The fix is to measure the noise before reacting to the signal. Repeating the suite under identical code, data, and configuration gives a direct read on run to run spread. That spread then sets the bar for what counts as a meaningful change. Small suites with model graded answers need this most, because judge variance adds a second noise source on top of generation variance.

Freeze everything except the randomness you intend to measure. Use the same model weights, same evaluation commit, same dataset snapshot, same generation limits, and same scorer version across repeats. Fix the data order or record the seed handling, since reshuffled few shot examples can shift scores without any model change.

Run the full suite three times for both the candidate and the baseline when a gate decision is close. Three is a small number, and it is used here because the assigned brief specifies three repeats as an illustrative procedure, not because three runs establish a tight confidence interval. Keep retrieval corpora, tool permissions, and sandbox images fixed across those runs.

Log the per run score, the per sample scores, and the environment digest for each repeat. The Inspect framework records eval logs with scores and metrics that support this kind of rerun analysis, and its scoring metrics documentation describes built in metrics such as accuracy, standard error, variance, and bootstrap standard error. The EleutherAI evaluation harness repository provides a widely used reference implementation for repeatable task definitions and result reporting.

Report the mean of the three runs alongside the range, which is the maximum minus the minimum. Also report each individual run score so reviewers can see whether one run is an outlier. For a 200 item suite with scores of 71.2, 72.8, and 70.9, the mean is 71.63 and the range is 1.9 points. Those three numbers are illustrative example values used to show the arithmetic, not results from a real experiment.

Add the standard deviation only when readers understand its limits at this sample size. With three observations the standard deviation is unstable, so prefer the plain range as the primary spread readout. Where tooling already computes standard error or bootstrap standard error, report the method name next to the number so later comparisons use the same estimator.

Keep the diagnostic separate from the headline metric. Dashboards should show the candidate mean, the baseline mean, and the measured run to run range in the same row. A gate that shows only green or red hides the margin information that determines whether the decision was close.

Use an explicit blocker rule that references measured spread. The brief for this article suggests treating a rollout blocker as valid only when the drop exceeds twice the measured run to run range. Treat that twice the range construction as an illustrative team policy, not an established statistical recommendation. It is a simple operational filter, not a significance test.

Apply it with a worked example using illustrative numbers. Suppose the baseline mean across three runs is 78.4 with a run to run range of 1.6, and the candidate mean is 75.9. The observed drop is 2.5 points. Twice the baseline range is 3.2 points. Since 2.5 is less than 3.2, the rule says do not block on this evidence alone; instead rerun, widen the sample, or inspect failing items. If the candidate mean were 74.6, the drop of 3.8 would exceed 3.2, and the blocker would stand pending item level review.

A workload derived alternative is to block when the candidate mean falls below the baseline mean minus the larger of two quantities: twice the baseline range, or a prespecified practical threshold such as 2 score points on a 100 point scale. The practical threshold prevents tiny but repeatable drops from blocking releases where the evaluation scale is coarse. Choose that threshold from product risk, document it, and apply it consistently.

Separate score variance into three buckets before assigning blame. Generation variance comes from sampling settings and nondeterministic execution. Scorer variance comes from exact match brittleness, judge disagreement, or floating point aggregation. Infrastructure variance comes from timeouts, truncated generations, rate limits, and cache differences.

Diagnose in a fixed order. First check completion rates and error counts per run, because, as an illustrative example, a run with 6 percent timeouts is not comparable to a clean run. Then check score distribution by item category to find whether the movement concentrates in one task type. Then inspect a sample of changed items by hand to see whether the model answer actually worsened or the scorer merely disagreed with a paraphrase.

Record temperature, top p, max tokens, judge model version, and seed policy in the release note. Without those five fields, a later team cannot reproduce the spread measurement and cannot tell whether a new drop reflects the model or a changed evaluation setting.

Repeat the closest gate comparison with fixed seeds where the stack allows it, then compare seeded spread against unseeded spread. If fixed seeds remove most of the movement, generation sampling dominates and wider sampling or a larger item count will stabilize the gate. If spread persists with fixed seeds, suspect scorer or infrastructure nondeterminism.

For model graded suites, grade the same transcripts twice with the judge and report judge only spread. When judge only spread explains most of the total spread, improve the grading rubric, add exact checks for deterministic items, or increase judge consistency controls before tightening the gate threshold.

Schedule the repeat runs with the same GPU type and the same parallelism settings, since changing batch size or tensor parallel layout can shift timing dependent behavior. Record queue wait per repeat so a repeat delayed behind training work is not misread as model variance.

Three repeats cannot detect small regressions. On a 500 item binary suite, a true 1 point accuracy drop sits well inside typical run to run movement, so the twice the range filter will correctly refuse to block while also missing a real but small change. Accept that tradeoff explicitly or increase the item count.

The range itself is noisy at this sample size. A lucky trio of close runs produces a narrow range and an aggressive gate, while an unlucky trio produces a wide range and a permissive gate. Reuse the historical range across recent releases rather than recomputing the bar from only the latest three runs.

Finally, variance auditing does not replace item level review. A stable mean can hide offsetting failures, where the model improves on common items and regresses on rare but important safety items. Always pair the spread check with a per category breakdown before clearing a rollout.

Inspect: Scoring metrics (accuracy, stderr, bootstrap, clustered errors)

EleutherAI lm-evaluation-harness repository

A GPU Fabric Benchmark Matrix That Fits in an Afternoon

Why Single-Pair Network Tests Overestimate Cluster Throughput