Define what reproduction means for this run
Reproducibility can mean that a job starts, that it reaches statistically comparable metrics, or that selected tensors match exactly. Write the target before collecting metadata. A debugging run may require the same first ten losses on one host. A research result may require a metric distribution across several seeds. An operational rerun may only need the same dataset, code, configuration, and acceptance range.
The target also states the boundary: GPU model, node count, driver branch, framework release, operating system, and allowed topology changes. PyTorch warns that complete reproducibility is not guaranteed across releases, commits, or platforms. Seeds reduce one source of variation; they do not turn different kernels, reduction orders, or hardware into the same computation.
Pin container content instead of recording a mutable label
Record the full image reference by manifest digest, registry, target platform, and the digest of every internally maintained base image. A tag such as training:release can move. The OCI image specification defines a descriptor digest as a content identifier derived from the referenced bytes. Preserve the resolved digest in the run record and in the job specification used by the scheduler.
Capture source revision, whether the source tree was dirty, build recipe revision, dependency lock files, compiler flags, and any package installed at job start. A container does not include the host kernel or GPU driver. It may also receive mounted libraries, plugins, secrets, and configuration at runtime. Inventory those inputs rather than claiming the image alone freezes the environment.
Record the driver, runtime, framework, and device chain
At startup, write machine-readable output for GPU identity, device UUIDs, driver version, CUDA runtime and toolkit where applicable, framework build, collective library version, and container runtime. NVIDIA documents that a CUDA application needs a driver compatible with the toolkit used to build it, with distinct backward, minor-version, and forward-compatibility cases. A version string that looks close is not proof of a supported combination.
Run a preflight that initializes every visible GPU, executes a small framework operation, forms the distributed process group, and performs a tiny collective. Fail before downloading a large dataset if this check fails. Store preflight output with the run. Compatibility proves the software can execute; it does not prove identical numerical behavior or performance.
| Layer | Stable identifier or evidence | Preflight |
|---|---|---|
| Container | Registry plus manifest digest | Pull and verify resolved digest |
| Host | OS, kernel, driver | Device query succeeds |
| Framework | Exact build and lock file | GPU tensor operation |
| Distributed stack | Library versions and config | Small collective |
| Hardware | GPU UUIDs and topology capture | All assigned devices visible |
Make the dataset and transformations addressable inputs
Record a dataset manifest version, storage namespace, shard names, byte sizes, and checksums. Include tokenizer files, vocabulary, normalization statistics, filtering rules, train-validation split, and any sampling weights. A bucket path or table name is only a location. If its contents can change, it cannot identify the training input.
Capture preprocessing code revision and configuration, including random augmentation settings. For generated or streaming data, record source offsets, query snapshot or export ID, generation model identifier, and retention policy. If privacy or storage policy prevents retaining raw data, say that exact reproduction expires when the snapshot does. A runbook cannot recreate bytes that were intentionally discarded.
Capture seeds, state, and the shape of execution
Record seeds for the application, framework, data sampler, augmentation libraries, and each distributed rank where separate streams are used. For a resume, save generator states and sampler position, not only initial seeds. Record world size, rank mapping, batch size per rank, gradient accumulation, precision mode, optimizer, scheduler, checkpoint step, and environment variables that change library behavior.
Execution order matters. Data-loader worker count, dropped final batches, elastic membership, collective algorithm selection, and asynchronous work can alter the order of floating-point operations. Store the scheduler job specification and topology capture. If exact replay is required, reject a rerun whose execution shape differs from the declared envelope instead of silently widening the promise.
Worked example: rebuild from an empty worker
For an illustrative drill, select run R-1842. Its record contains image digest sha256:example-placeholder, source commit, dependency lock checksum, driver compatibility requirement, eight device UUIDs, dataset manifest D-77, tokenizer checksum, job configuration C-19, five seeds, and checkpoint K-42000. The placeholder is an example format, not a real image identifier. Provision a clean supported worker and allow the runbook to retrieve only those addressed artifacts.
The drill passes startup when every checksum resolves, the actual image digest matches, preflight completes on all assigned GPUs, and the configuration hash is C-19. It passes behavioral reproduction when the next 100 steps remain inside a tolerance fixed before the drill and the evaluation metric lands inside the run's predeclared seed-aware interval. If the contract demands bitwise equality, one unequal selected tensor is a failure, even if the final score is close.
Operate the runbook as a release gate
Require a reproduction record before a run may be promoted as a baseline. On every framework, driver, base-image, or dataset change, rebuild one pinned reference run on a clean worker. Diagnose failures in order: artifact resolution, digest mismatch, device and driver preflight, dataset checksum, configuration difference, first divergent batch, first divergent model state, then later metric drift. This sequence narrows the search boundary.
Retain the record, small diagnostic inputs, evaluation outputs, and at least one restorable checkpoint for the declared lifetime. Review a sample monthly and before retiring a registry or dataset snapshot. Capacity sourcing can honor the run only after the record states the supported device, topology, storage, and driver constraints; procurement cannot replace missing provenance.
Publish the reproducibility limit with the result
State whether the run was replayed, on what hardware and software envelope, how many seeds were evaluated, which outputs were compared, and what tolerance passed. Do not describe a single matching metric as deterministic reproduction. Statistical agreement needs a specified test and enough runs to support it, while exact replay needs a much narrower environment.
This runbook reduces accidental variation and makes differences diagnosable. It cannot guarantee identical results across unsupported hardware, releases, or nondeterministic operations, and it cannot preserve an unavailable external service. The honest deliverable is a bounded claim with durable inputs and a test, not an assertion that every future machine will emit the same bits.