Price the two kinds of waste
A checkpoint interval trades recurring write overhead against work lost after a failure. Saving too often pauses or slows useful training and creates more storage traffic. Saving too rarely makes each interruption erase more completed work. The useful unit is not checkpoints per hour. It is expected cost per productive hour, with compute waste and storage consumption expressed separately before anyone combines them into money.
Start with four measured inputs: C, the elapsed time added by one checkpoint; M, the mean time between job-ending failures under the relevant topology; R, the restore and restart time; and S, the durable bytes per checkpoint. Treat M as an operating estimate, not a hardware promise. A 256-GPU job can have a different interruption process from a single node because its software, network, scheduler, and storage dependencies are different.
Derive a first-pass interval
For a first-order stationary model, let T mean productive work time between completed checkpoints, let failures be independent with mean time to interruption M, and assume T and checkpoint time C are small relative to M. A failure loses about T/2 of productive work on average, while checkpointing costs about C/T of productive time. Ignoring higher-order terms and terms independent of T gives the waste approximation C/T + T/(2M). Differentiating gives T = sqrt(2CM). This is the Young first-order result, not a guarantee of the true optimum. Some sources define checkpoint period to include C, so interval labels must state which convention they use.
Restore time R affects total availability even though a constant R drops out of this first-order derivative. The corresponding planning approximation is C/T + T/(2M) + R/M. It ignores failures during checkpointing and recovery, as well as the small distinction between productive time and wall time. A long restore makes checkpoint validity and placement more valuable, but does not by itself change the minimizing T under these assumptions. If restore time varies with checkpoint age or storage tier, include that dependence when comparing candidate intervals.
Work an illustrative case
Consider an illustrative job where a checkpoint adds C = 6 minutes and observed job-ending mean time between failures is M = 72 hours. Convert both to minutes: M = 4,320. The first pass is sqrt(2 x 6 x 4,320) = sqrt(51,840) = about 228 minutes, or 3.8 hours. If restore and relaunch take an illustrative 25 minutes, expected restore loss is 25/4,320 = 0.58 percent of elapsed time.
At T = 228 minutes, checkpoint overhead is 6/228 = 2.63 percent and expected recomputation loss is 228/(2 x 4,320) = 2.64 percent. The variable loss is about 5.27 percent, plus the 0.58 percent restore term. These scenario numbers are illustrative, not measured market prices or performance. Actual failure rates, write times, storage prices, and compute prices vary, so teams should replace every input.
| Candidate interval | Checkpoint loss | Expected recompute loss | Variable total |
|---|---|---|---|
| 120 min | 5.00% | 1.39% | 6.39% |
| 228 min | 2.63% | 2.64% | 5.27% |
| 360 min | 1.67% | 4.17% | 5.84% |
Add storage without confusing flow and stock
The interval controls write flow, while retention controls stored stock. If S = an illustrative 800 GB and the job runs continuously, T = 228 minutes produces about 6.32 checkpoints per day, or 5.05 TB of new logical data per day before replication, compression, deduplication, and deletion. Keeping every checkpoint for seven days would expose roughly 35.4 TB. Actual storage accounting varies, and the example is capacity arithmetic rather than a provider quote.
Use a constraint before a blended dollar formula: reject any interval whose write rate exceeds the sustained checkpoint path budget or whose retention policy exceeds the allocated capacity. Among feasible intervals, compare expected compute-loss cost plus incremental storage and I/O cost. Any monetary conversion is hypothetical until it uses the workload's own measured GPU-hour, storage, and I/O prices; the interval itself should come from measured C, M, R, and S.
Recalculate when failure risk is not stationary
The simple formula assumes a constant failure hazard and similar interruptions. Real jobs can violate both assumptions. Preemptible capacity may have scheduled notices, a flaky node may create a burst of failures, and maintenance windows may change risk by time of day. Pooling those periods into one M hides structure. Segment observations by instance class, cluster, job scale, software release, and operational window, then calculate intervals for segments with enough exposure.
Define a local review trigger rather than treating any percentage as universal. For example, recompute when either the rolling failure rate or checkpoint duration moves by more than an illustrative 20 percent from the policy input, provided the window contains enough exposed job-hours and failures to report uncertainty rather than reacting to one event. When a termination notice is reliable, an event-triggered checkpoint can complement the periodic schedule. When failures cluster, use survival curves or replay historical traces instead of assuming a constant hazard.
Validate with recovery receipts
Run the decision as a diagnostic sequence. Measure checkpoint wall time at steady load, count bytes and storage throttling, record job-ending failures per exposed job-hour, and time restore through the first successful post-restore training step. Evaluate three or more candidate intervals against the same trace. Then conduct a controlled recovery test that loads the artifact, restores optimizer and scheduler state, and checks the resumed step and loss for plausibility.
Adopt the longest candidate whose expected total loss is within 0.5 percentage points of the minimum and that satisfies the recovery point objective and storage constraints. That tolerance avoids operational churn for a mathematically tiny gain. The limitation is model risk: correlated outages, deterministic application bugs, validation failures, and queue delays can dominate the average-case equation. Keep raw event data and review the rule after topology or checkpoint format changes.
Report uncertainty beside the chosen interval. A mean time between failures estimated from a short run can move substantially when the next event arrives. Recalculate the candidate interval using a plausible low and high value for M, then check whether the operational choice changes. If 2.5, 3, and 4 hours have nearly identical expected cost across that range, prefer the interval that is easier on storage. If the choice swings widely, collect more exposure before treating the computed optimum as precise.