Freezing evaluation prompts and seeds for comparable release gates

Release gates often compare numbers that were not produced under the same conditions. A prompt template gains a clarifying sentence, a sampling temperature is adjusted in a serving config, or an answer parser is fixed to accept a new format. The model weights may be identical, yet the reported accuracy moves by a point or two. Teams then debate whether the candidate improved or whether the test changed.

A frozen evaluation digest addresses this by naming the exact test conditions for each candidate. The lm-evaluation-harness project follows the same idea by defining tasks, prompts, and metrics in versioned task configuration files so results can be compared across runs. When prompts, seeds, and scoring code are pinned to one digest, a later change is visible before anyone approves a promotion.

A workable digest covers three inputs and nothing else by default. First, the prompt pack, which includes system instructions, user templates, few shot examples in order, and decoding settings such as max tokens and stop strings. Second, the seed set, which includes sampling seeds, data order seeds, and few shot selection seeds. Third, the scoring code, which includes answer extraction, text normalization, metric code, and library versions.

Each input is hashed separately with SHA256, then the three hashes are concatenated and hashed again to form the release digest. As a format illustration with placeholder prefixes, if prompt hash starts with 9c41, seed hash starts with 77be, and scoring hash starts with b205, the digest is SHA256 of that combined string. Store the digest alongside the model checkpoint identifier, the dataset snapshot identifier, and the date of the run.

Keep the digest small enough to check in a gate script. A JSON record with six fields is enough: candidate name, model checkpoint hash, prompt pack hash, seed set hash, scoring code hash, and final digest. Any reviewer can recompute the digest from the stored files and confirm that the reported score belongs to those exact inputs.

Prompts cause the most silent drift because edits look harmless. Adding please reason step by step, changing a system role from assistant to expert assistant, or reordering two examples can alter outputs for sampled tasks. The Hugging Face GenerationConfig documentation lists decoding controls such as temperature, top p, top k, and max new tokens as separate parameters, which shows why prompts and decoding settings must be frozen together rather than reviewed separately.

Treat prompts as data files under version control. Store each template as a text file with a version number, keep few shot examples in a CSV or JSONL file with fixed ordering, and forbid inline prompt strings in serving or evaluation scripts. A pull request that changes a comma in a template should change the prompt hash, which is the intended behavior.

Review prompt diffs with the same care as code diffs. Ask whether the edit changes task difficulty, gives away part of the answer, or alters output format in a way the scorer rewards. If any answer is yes or uncertain, keep the old pack for the current gate and schedule the new pack for a full re-baseline.

Randomness enters evaluation in at least three places: token sampling during generation, shuffling or subsampling of test items, and selection of few shot examples. PyTorch documentation states that torch.manual_seed seeds the random number generator for all devices including CPU and CUDA, and that Python random and NumPy generators need separate seeding. The same page states that fully reproducible results are not guaranteed across releases, commits, or platforms, or between CPU and GPU execution.

That distinction matters for gate design. Setting seeds reduces run to run variation on the same machine and software stack, which is a useful diagnostic control. It does not promise bitwise identical outputs after a driver upgrade, a library change, or a move to different GPU hardware. Treat equal seeds as a way to make comparisons fair, not as proof that hardware effects are gone.

Record every seed that affects the gate. Include the sampling seed per prompt, the seed used to order test items, the seed used to pick few shot examples, and DataLoader generator settings where data loading is involved. PyTorch recommends disabling cuDNN benchmarking and setting deterministic flags where available when repeatability matters, with the warning that speed may drop. For release gates, prefer a small fixed seed set such as three recorded seeds and report the mean and range rather than a single lucky draw.

Scoring code decides what counts as correct, so it belongs inside the digest. Answer extraction often uses regular expressions, unit normalization, whitespace handling, or multiple choice letter parsing. A fix that strips dollar signs or accepts a new date format can raise scores without any model change. Pin the scorer file, the metric definition, and the versions of scoring libraries together.

Store per item outputs alongside the aggregate score. For each test item, keep the prompt identifier, the sampled output, the extracted answer, and the item score. When a scorer changes, the team can rescore stored outputs with the new code and separate parser effects from model effects. This stored evidence also supports audits when a later release questions an old promotion.

Consider an illustrative release gate with invented numbers used only to show arithmetic. Candidate rc-2026-10-09 ships with prompt pack v14, seed set s3, and scorer v7. The three SHA256 hashes combine into digest 4f2a9c. Baseline model rc-2026-09-28 was scored at 71.4 percent on 800 items with the same digest. The gate compares equal digests, so the candidate score of 72.1 percent can be read as a 0.7 point move on the same test.

Before promotion, an engineer fixes a typo in one template and reruns only the candidate. The prompt hash changes, so the digest becomes 8d11e0 and no longer matches the baseline digest. The gate script blocks promotion and asks for a recorded re-baseline. The team then scores the previous model with the corrected template and records 71.9 percent, which shows that 0.5 points of the apparent gain came from the template edit.

The remaining candidate gain is 0.2 points on 800 items. With three recorded seeds, the candidate range is 71.8 to 72.4 and the rescored baseline range is 71.5 to 72.0. The intervals overlap, so the team holds the promotion for more items rather than claiming a win. All numbers in this paragraph are illustrative and actual results vary by task and model.

Operate the gate with a simple rule that anyone can check. Promotion is allowed when the candidate digest equals the baseline digest and the candidate mean exceeds the baseline mean by more than twice the pooled seed range. Otherwise promotion is blocked until a re-baseline is recorded. The twice range threshold is an illustrative team policy, not an established recommendation, and teams should derive their own margin from past seed variation on their own tasks.

A frozen gate improves comparability but has clear limits. It does not predict behavior on new production traffic, it does not remove hardware or library nondeterminism, and it cannot catch data contamination if test items leaked into training. When the frozen suite needs a full re-baseline, ClusterBid can help match rental GPU capacity to the measured job length so the repeat run finishes without queue delay, with actual rental prices varying and no price quoted here. Keep the frozen suite narrow, review failures by hand, and update the pack only with a recorded re-baseline for both candidates.

EleutherAI lm-evaluation-harness repository

PyTorch Reproducibility notes

Hugging Face Transformers GenerationConfig and generate documentation

Before Scaling Training: A One-Node Performance Acceptance Test

A GPU Capacity Runbook for Product Launch Days