Use completed work as the denominator
A lower GPU-hour rate does not guarantee cheaper training. Interrupted time can include computation after the last checkpoint, checkpoint writing, shutdown, time waiting for replacement capacity, image and dataset loading, checkpoint restoration, and warm-up before measured throughput returns. The useful output is accepted optimizer steps or training tokens that survive into a completed run. Divide every attributable cost by that output.
Keep wall-clock delay and billed cost separate. A requeue wait may add no GPU bill but can miss a deadline. Storage requests and retained checkpoints may add cost without using a GPU. A failed restore can consume both. The comparison should report effective cost per million accepted tokens, completion time, and probability of meeting the deadline. A single discount percentage hides all three.
Build the model from an interruption ledger
Instrument every attempt with start time, first productive step, checkpoint times, interruption notice time, last committed step, termination time, requeue entry and exit, restore completion, and first productive step after recovery. Mark whether the notice was observed and whether an emergency checkpoint committed. AWS states that its Spot interruption notices are best effort and generally provide a two-minute warning, so a cost model should not assume every event allows a final save.
Classify time into productive compute, doomed compute, checkpoint pause, reload and warm-up, and idle wait. Classify money into GPU runtime, attached CPU and memory where billed, durable storage capacity, storage operations, and transfer charges that actually apply. Do not assign a generic cloud fee without a provider invoice or calculator. The arithmetic below uses explicitly hypothetical planning inputs, and actual prices vary by provider, region, contract, and date.
Worked example: a large discount becomes a small one
Assume an illustrative single-GPU-equivalent job needs 100 productive GPU-hours. The stable-capacity baseline is assumed to bill exactly those 100 hours, including any routine checkpointing already present in both cases. At a hypothetical $4.00 per GPU-hour, its compute cost is 100 x $4.00 = $400. Assume spot capacity at a hypothetical $1.60 per GPU-hour. Across the spot run, interruptions cause 18 GPU-hours of lost work, additional interruption-related checkpoint pauses consume 4 GPU-hours, and reload plus warm-up consumes 8 GPU-hours. Total billed spot runtime is 100 + 18 + 4 + 8 = 130 GPU-hours, costing 130 x $1.60 = $208.
Add an illustrative $22 for incremental checkpoint storage and operations and $10 for failed-attempt data loading. Effective spot cost is $208 + $22 + $10 = $240, or $2.40 per productive GPU-hour. That is 40 percent below the assumed $400 baseline, not the 60 percent suggested by raw rates. For this single-GPU-equivalent example, if the run also waits 15 unbilled hours for capacity, cost stays $240 but completion time grows from 100 hours to 145 hours. For a multi-GPU job, GPU-hours must be converted to wall-clock time using the active GPU count rather than added directly. Every dollar figure here is hypothetical; actual prices vary.
| Component | Illustrative quantity | Illustrative cost |
|---|---|---|
| Productive runtime | 100 GPU-h x $1.60 | $160 |
| Lost work | 18 GPU-h x $1.60 | $28.80 |
| Checkpoint pause | 4 GPU-h x $1.60 | $6.40 |
| Reload and warm-up | 8 GPU-h x $1.60 | $12.80 |
| Storage and loading | Assumed additions | $32 |
| Total | 130 billed GPU-h plus additions | $240 |
Solve for the overhead ceiling before launching
Let D be spot rate divided by stable rate, O be nonproductive billed GPU-hours divided by productive GPU-hours, and A be extra non-GPU cost per productive GPU-hour divided by the stable rate. Ignoring deadline value, spot wins when D x (1 + O) + A is less than 1. This exposes the maximum tolerable overhead instead of treating interruption frequency as an isolated metric.
Using the illustrative rates above, D = 1.60 / 4.00 = 0.40. If added storage and loading average $0.32 per productive GPU-hour, A = 0.32 / 4.00 = 0.08. The ceiling is O < (1 - 0.08) / 0.40 - 1 = 1.30, or 130 percent. With a smaller hypothetical discount, $3.00 versus $4.00 and the same A, the ceiling becomes (1 - 0.08) / 0.75 - 1 = 22.7 percent. Actual prices vary, and teams should substitute measured rates and charges.
Promote spot only after a shadow cost gate
Require enough observations to cover several checkpoint cycles and interruption events. A practical gate is: effective cost per accepted token must remain at least 15 percent below the stable pool at the upper confidence bound, p95 completion time must meet the job deadline, and clean restore success must exceed the service target. The 15 percent buffer is a planning choice, not an industry constant. It absorbs unmodeled operational effort and rate movement.
If no interruptions occur during a short pilot, do not infer zero overhead. Inject controlled termination into a disposable run, measure reload and restart costs, and evaluate a range of interruption intervals. Compare candidate pools using the job's measured restore time, topology constraint, and deadline, not the posted GPU rate alone.
State what the spreadsheet cannot settle
The model cannot price every consequence. Engineer attention, experiment scheduling, lost debugging context, and a delayed model release may dominate infrastructure charges but lack a stable dollar value. Keep them as separate risk indicators instead of inventing precise costs. Stable capacity can also fail, so its baseline should include observed failure and maintenance overhead rather than assume perfection.
Recompute after changes to model size, checkpoint format, dataset location, image size, world size, or capacity pool. Stop using spot for a job class when the rolling effective-cost advantage falls below the chosen buffer or its completion-time objective fails twice in the review window. The purpose is not to reject interruptions. It is to buy completed work at a known price and delay.