Budget a campaign, not one successful run
A training estimate built from one clean run answers the wrong question. A campaign includes short feasibility experiments, data-pipeline validation, hyperparameter trials, interrupted jobs, resumed jobs, periodic evaluation, final evaluation, checkpoint writes, artifact retention, and analysis. Define the unit of progress first: accepted optimizer steps, processed tokens, or validated samples. Then attach GPU-hours and storage to each campaign activity rather than multiplying a single runtime by an hourly rate.
Keep planned work and contingency visible as separate lines. Planned evaluation is not failure overhead, and failed runs are not automatically waste because early experiments can eliminate poor configurations. The budget should show what decision each job supports. This discourages a pile of untracked exploratory jobs while preserving room for research. It also makes actual-versus-plan review possible when iteration count, model shape, or data volume changes.
| Work class | Primary driver | Budget evidence |
|---|---|---|
| Pilot | Candidate count and duration | Experiment plan |
| Main training | GPU count and useful runtime | Step-time sample |
| Retries | Failure timing and recovery | Job history |
| Evaluation | Checkpoint count and suites | Evaluation schedule |
| Storage | Artifact size and retention | Manifest and policy |
Separate useful, failed, and recovery hours
For every job, record allocated GPU-hours, useful GPU-hours before the stopping point, and recovery GPU-hours needed to regain that point. Allocated hours equal GPU count multiplied by elapsed allocation time. Useful hours depend on accepted progress, not merely device activity. A failed run may still produce a reusable checkpoint, while a successful process exit may produce an invalid result. Store scheduler state and exit code alongside application progress and correctness.
Slurm sacct can report elapsed time, state, exit code, requested resources, allocated resources, and resource usage fields. That makes it a useful source for allocation accounting, but it cannot decide which steps were scientifically useful. Join scheduler records to experiment identifiers, code revision, data snapshot, and accepted checkpoint. If the scheduler's clock and the application log disagree, preserve both and investigate rather than selecting the cheaper number.
Price evaluation as real compute
Evaluation can include loss passes, generation, benchmark harnesses, safety tests, regression suites, and human-review sampling. Some run on every checkpoint; others run only on promoted candidates. Specify the number of checkpoints, suites per checkpoint, replicas or decoding samples, GPU count, and elapsed time. Include model loading and result writing when they occupy the allocation. A training job that fits eight GPUs does not imply every evaluation should reserve eight.
PyTorch Profiler can collect CPU and device activity, operator timing, shapes, memory information, and traces. Use a scheduled, representative profile to diagnose evaluation work rather than profiling the entire campaign by default. Profiling overhead and trace storage belong in the diagnostic budget. Evaluate correctness with task-specific measures, not utilization. A GPU can be fully active generating redundant samples, and a low-utilization evaluation may still be required for a release decision.
Work a campaign scenario line by line
Illustrative assumptions only: six pilot runs use 4 GPUs for 3 hours each, one main run uses 8 GPUs for 120 hours, two retry attempts each use 8 GPUs for 10 hours, and 12 evaluation jobs use 2 GPUs for 2.5 hours. GPU-hours are pilots 6 x 4 x 3 = 72, main 8 x 120 = 960, retries 2 x 8 x 10 = 160, and evaluation 12 x 2 x 2.5 = 60. Total planned consumption is 1,252 GPU-hours.
At a hypothetical $2.40 per GPU-hour, compute would be 1,252 x $2.40 = $3,004.80. Suppose 18 retained checkpoints average 180 GB and are kept for two months. Storage exposure is 18 x 180 GB x 2 = 6,480 GB-months before replicas or version overhead. Apply a separately documented storage assumption to that quantity. These figures are illustrative, not quotes or performance claims; actual prices, runtimes, compression, checkpoint content, and billing units vary.
Test failure and iteration sensitivity
A base case is incomplete without sensitivity. Let P be pilot GPU-hours, T main-run GPU-hours, E evaluation GPU-hours, f be additional failed and recovery GPU-hours expressed as a fraction of T after accounting for usable checkpoints, and n be the number of complete main attempts. A simple campaign model is P + nT(1 + f) + E. This sensitivity model replaces the explicit retry line in the preceding base example; it must not be added on top of that line. It is deliberately transparent. More detailed models can place failures over time and distinguish restart overhead from recomputation.
Using P = 72, T = 960, E = 60, and n = 1, a 5 percent loss gives 72 + 960 x 1.05 + 60 = 1,140 GPU-hours. A 20 percent loss gives 1,284 GPU-hours, 144 more. If scientific review requires a second full main attempt, the 5 percent case becomes 72 + 2 x 960 x 1.05 + 60 = 2,148 GPU-hours. Iteration count dominates small rate changes, so approval should specify both a spend ceiling and the decisions that unlock another attempt.
Treat checkpoints as recovery assets with costs
Checkpoint frequency trades storage and write overhead against recomputation after interruption. Include model state, optimizer state, scheduler state, random-number state where needed, and data position required for a faithful resume. Test restoration before depending on it. PyTorch's serialization documentation explains that torch.save and torch.load serialize tensors and objects, while torch.load also warns about untrusted inputs. Operationally, keep provenance and access controls with the artifact rather than treating a checkpoint as an anonymous file.
Retain checkpoints according to decisions: recent recovery points, promoted candidates, and final reproducibility artifacts may need different periods. Measure full checkpoint size and write duration with the intended distributed format and storage path. Activation checkpointing is a different mechanism that recomputes forward segments during backward execution to save memory; it should not be confused with durable training-state checkpoints. The first changes runtime and memory; the second enables recovery and consumes persistent storage.
Use a measurable budget control loop
Before launch, set gates for pilot promotion, main-run continuation, retry authorization, evaluation promotion, and retention. One possible rule is: promote only candidates that meet the correctness floor; stop when projected campaign GPU-hours exceed the approved ceiling; authorize a retry only when the diagnosed cause is addressed or the expected recovered value exceeds its cost; run expensive evaluation only on checkpoints passing the cheaper suite. These rules should be adapted to the research goal rather than copied mechanically.
Review weekly using job records and experiment outcomes. Diagnose variance in this order: reconcile allocated hours, classify failed and canceled jobs, verify recovered versus recomputed steps, count evaluation invocations, then reconcile retained artifacts. Use the resulting measured GPU shape and schedule when comparing capacity options. No budget model predicts discovery: promising results may justify more iterations, while invalid assumptions may end a campaign early. Make that uncertainty explicit with scenarios and approval gates instead of hiding it inside a flat contingency percentage.