Define which repetition is a defect
Training normally revisits examples across epochs, so duplicate does not simply mean an identifier appears twice in a run. The restart defect is unintended replay within the logical data schedule, usually because the model resumes from one optimizer boundary while the input pipeline resumes from an earlier point. Another defect is omission, where prefetched examples are counted as consumed even though their gradients were never committed.
Choose a progress unit that corresponds to training semantics. For fixed examples it may be a stable sample ID. For packed language-model batches it may be source-document offsets plus valid-token counts. For streaming data it may be partition and offset. Record the unit at the optimizer commit boundary. A DataLoader batch handed to a worker is not necessarily trained, and a completed backward pass is not necessarily an optimizer update when gradient accumulation is active.
Sampler state is more than an epoch number
PyTorch DistributedSampler assumes a constant-size dataset whose elements keep a stable order. With shuffling enabled, it creates a deterministic permutation from the common seed and epoch. The documentation requires set_epoch at the beginning of each epoch so different epochs receive different orderings. Restoring only the seed without the epoch can repeat an earlier permutation, while restoring only the epoch restarts that epoch from its beginning.
The saved state should identify dataset version, sampler algorithm and configuration, shuffle seed, epoch, global cursor within that epoch, replica count, drop-last choice, and batch construction rules. Persist enough information to rebuild the order or store an explicit manifest. If augmentations use separate random generators, save their state or accept that the same source sample may receive a different transform after restart. That is a reproducibility issue distinct from source-sample duplication.
World size changes the partition boundary
DistributedSampler pads indices when drop_last is false so every replica receives the same count, or removes a tail when drop_last is true. Its source then slices the global index list by rank. Changing the replica count changes total padding and the slice assigned to each rank. A rank-local cursor such as rank 3, batch 900 cannot be translated reliably into a new world size without reconstructing the global ordering.
Resume from a global cursor before repartitioning. Rebuild the epoch permutation for the saved dataset version, advance past globally committed units, then divide the remainder among the current ranks. For iterable datasets, ownership must be explicit because DataLoader cannot infer whether worker replicas overlap. Partition offsets or shard manifests are safer than a count of yielded batches, especially with variable batch sizes.
Gradient accumulation creates the commit boundary
Suppose one optimizer update contains four microbatches. A failure after the third backward pass leaves gradients in volatile memory, but the durable model still reflects the preceding optimizer update. If the checkpoint says those three microbatches were consumed, resuming at the fourth omits their contribution. If it says none were consumed, replaying all four is correct for the model state even though three were previously read and computed.
The clean policy is to checkpoint only after optimizer.step, scheduler advancement, scaler update, and sampler commit are complete. If mid-accumulation checkpoints are required, save accumulated gradients, the accumulation index, loss-scaler state, and exact microbatch identities as one transaction. This is harder to validate and often larger. Prefer the optimizer boundary unless recovery objectives justify the complexity.
A worked ledger exposes replay
Consider an illustrative run with eight data-parallel ranks, microbatch 2 per rank, and four accumulation steps. Each optimizer update commits 8 x 2 x 4 = 64 samples. The durable checkpoint records update 1,250, so it represents 1,250 x 64 = 80,000 committed sample positions. A crash during the third microbatch of the next update does not move that durable cursor. Resume at position 80,000 and rebuild the entire next 64-sample update.
Now assume the input log shows 80,048 samples fetched before failure. The 48-sample difference is abandoned speculative work, not a training duplicate when replayed, because no matching optimizer update was committed. This distinction prevents false alarms. The numbers are illustrative. If an optimizer can partially update parameters or checkpointing is not atomic across ranks, the simple ledger is insufficient and the incomplete generation must be rejected.
| Counter | Illustrative value | Meaning |
|---|---|---|
| Fetched positions | 80,048 | May include uncommitted prefetch or accumulation |
| Committed positions | 80,000 | Represented by durable optimizer state |
| Resume position | 80,000 | Safe boundary for model and data state |
Audit the restart boundary with identifiers
Emit a compact audit record for a window before and after each checkpoint: optimizer update, accumulation slot, rank, sample or token-range identifier, dataset version, and committed flag. A hash of ordered identifiers can reduce volume, but retain a small readable sample for diagnosis. On restart, compare the reconstructed first window with the expected suffix of the global order before allowing a long run.
Use a diagnostic sequence: verify the checkpoint’s committed cursor, regenerate the next identifiers without loading tensors, compare them to the manifest, run one optimizer update, and confirm the cursor advances by the expected global amount. Reject the resume if a committed position reappears before the intended next epoch or if the gap exceeds the documented padding policy. Compare positions as well as source IDs because DistributedSampler may intentionally repeat source indices when drop_last is false and padding is required. Bloom filters can flag likely repeats at scale, but false positives mean they should trigger investigation rather than automatic deletion.
Choose and disclose an acceptable deviation
Exact once-only ordering is not always worth its coordination cost. Some training programs accept replay of at most one global batch after a failure, particularly when datasets are huge and failures are rare. Others cannot tolerate biased replay of a small high-weight subset. State the bound in samples or tokens, not vague language, and measure observed duplicate and omission rates by dataset stratum.
A practical rule could require zero replay of committed positions, zero unexplained omissions, and no more than one uncommitted global batch recomputed after a crash. That rule is measurable and aligns model state with data state. It does not guarantee identical final weights: different worker timing, augmentations, floating-point reduction order, and a changed world size can still alter the trajectory. Preserve the validated sampler, dataset version, and batch contract when moving the run. The goal is a defensible data ledger, not a promise of bitwise reproduction.