Retain a recoverable unit, not a filename
A checkpoint is often a set of shards plus metadata, not one interchangeable file. Recovery may also depend on the model configuration, tokenizer, optimizer layout, random-number state, data cursor, and code or container identity. Retention must reason about that dependency set. Deleting one shard because its modification time looks old can leave a directory that appears present but cannot restore a distributed job.
Publish a manifest for each checkpoint generation. It should name every required object, byte size, checksum, logical training step, world-size or sharding assumptions, and the immutable identifiers needed to recreate the runtime. Write data under a temporary generation identifier, verify all members, then publish a small completion marker or committed manifest last. Only committed generations enter the retention inventory.
Separate hot, warm, and anchor copies
A useful policy has roles rather than one age cutoff. Hot checkpoints optimize rapid retry and remain on the fastest shared tier. Warm checkpoints cover a longer rollback window on less scarce capacity. Anchor checkpoints mark milestones such as a validated evaluation or data boundary and may live longer. The exact tiers can be directories, storage classes, or distinct systems, but their failure domains should be understood.
An illustrative policy could keep the newest four committed checkpoints hot, one checkpoint per six hours for two days warm, and one validated daily anchor for seven days. If each role is materialized as a separate 600 GB copy, the conservative upper bound is 4 x 600 GB + 8 x 600 GB + 7 x 600 GB = 11.4 TB. If roles are labels on one generation, deduplicate overlaps and compute the union instead; it will be no greater than 19 generations. Replication, versions, and metadata can increase physical use. These sizes are hypothetical, and actual storage behavior and prices vary.
| Role | Illustrative retention | Deletion prerequisite |
|---|---|---|
| Hot retry | Newest 4 | Newer generation restored successfully |
| Warm rollback | Every 6 hours for 2 days | Anchor exists and manifest verifies |
| Anchor | Daily for 7 days | Policy age met and no active hold |
Make validation a deletion gate
A successful upload does not prove a successful recovery. First validate the manifest, object count, sizes, and checksums. Next perform a format-level load in an isolated process. On a schedule and after checkpoint code changes, run a full restore on a representative topology through at least one training step. Record the generation, validator version, result, and timestamp as an immutable receipt.
Garbage collection should require three facts: the candidate is committed, a newer generation has a valid restore receipt, and no active job or incident hold references it. If any fact is unknown, retain the candidate and alert. This conservative rule temporarily uses more capacity, but it prevents a cleanup race from turning silent checkpoint corruption into total recovery loss.
Budget capacity with two watermarks
Give each project a soft quota and a hard safety boundary. At the soft watermark, stop admitting extra non-anchor retention and accelerate policy-compliant collection. At the hard watermark, fail new checkpoint admission clearly or redirect it to an approved spill tier. Do not let arbitrary writes consume the reserve required to complete an in-flight generation, because a nearly full filesystem can strand both the old and new checkpoints.
Size the reserve from measured peak generation size plus temporary overhead. For a hypothetical 600 GB generation written through a temporary directory, retaining one old copy while creating and validating the next requires 1.2 TB for checkpoint data, plus filesystem and manifest overhead. Set the high watermark below total capacity by the full measured reserve. This arithmetic is illustrative; actual allocation, replication, sparse-file behavior, and storage accounting vary.
Collect with a mark-and-sweep pass
Build the live set first. Mark all checkpoint generations selected by hot, warm, and anchor rules, plus the newest committed generation, active readers, incident holds, and dependencies referenced by manifests. Take a versioned snapshot of that decision. Sweep only unmarked objects whose generation was already visible at snapshot time. This makes a concurrently arriving checkpoint ineligible for accidental deletion.
Use tombstones or a quarantine prefix before irreversible deletion when the backend permits it. Record candidate bytes, policy reason, manifest identity, and final outcome. A measurable decision rule is to start collection at 75 percent allocated usage and finish before 85 percent, while always preserving the measured completion reserve. If the collector cannot prove it can reach the target, reject new jobs rather than deleting anchors.
Treat incomplete uploads as a separate inventory
Incomplete multipart uploads may consume storage without appearing as completed objects. Amazon S3 documentation states that uploaded parts remain stored until the multipart upload is completed or aborted, and lifecycle expiration of objects is not the same as aborting incomplete uploads. Track pending upload identifiers, initiation times, owners, and bytes separately from committed checkpoint generations.
Set an application cleanup threshold longer than the longest measured legitimate upload plus retry margin. For example, if the p99 checkpoint upload completes in a hypothetical 50 minutes, a four-hour application threshold gives room for retries. For Amazon S3, the lifecycle AbortIncompleteMultipartUpload action is configured in whole days after initiation, so it is a later safety net rather than a four-hour mechanism. Never abort solely by filename age while a writer lease is live. AWS also notes that part operations already in flight can finish after an abort request, so wait for writers to stop and verify that the upload and its parts are gone.
Prove the policy under pressure
Test in order: create a complete generation, interrupt another midway, hold an old anchor, start a reader, cross the soft watermark, and run collection. Confirm that the partial upload is eventually reclaimed, the reader and hold survive, the newest valid generation restores, and usage falls below target. Export counts for committed, validating, failed, quarantined, and incomplete generations so hidden bytes become visible.
The main limitation is that metadata correctness cannot compensate for a shared failure domain. If every retained generation and its manifests sit on one array, an array-wide loss defeats the retention policy. Keep at least one tested recovery copy in an appropriately independent domain. Retention also cannot infer application semantics, so milestone anchors and legal or research holds require explicit owner input rather than automatic age rules.
Review the receipts as an inventory reconciliation, not just a dashboard. Sum bytes from committed manifests, temporary generations, quarantine, and backend-reported incomplete uploads, then compare that total with quota accounting. A persistent gap signals untracked versions, snapshots, deleted-but-open files, replication overhead, or an accounting delay. Set an owner and deadline for unexplained bytes. The policy is working only when operators can connect capacity consumption to a recoverable generation or a bounded cleanup state.