Reporting Training Throughput During Checkpoint Windows

Active-step throughput measures work completed while training steps are executing. It helps diagnose kernels, input stalls, and collective communication. Sustained throughput measures committed samples or valid tokens divided by elapsed wall-clock time across a representative interval that includes scheduled checkpointing, validation, and ordinary control-plane pauses. It answers the planning question: how quickly does the run actually progress toward its training target?

Publishing only the active rate can make a system with long save windows look equivalent to one that keeps progressing. Publishing only the sustained rate can hide whether the loss comes from checkpoint I/O, evaluation, or slow compute. Keep both, name their denominators, and never average per-step rates to approximate the end-to-end rate. The correct sustained calculation divides total useful work by total elapsed time.

A benchmark window should start and end at comparable semantic boundaries, such as immediately after two durable checkpoint commits, and contain the normal checkpoint and validation cadence. A five-minute sample taken between hourly saves is useful for compute diagnosis but is not a capacity forecast. Warm-up, compilation, cache fill, and startup may be reported separately, provided the exclusion is explicit and relevant to the intended long-running workload.

MLCommons training rules define timestamped run-start and run-stop events and specify benchmark-specific quality targets, making the measured boundary auditable. An internal benchmark does not inherit those rules merely by citing them. It should define its own contiguous wall-clock boundary and declare whether checkpoint writes, asynchronous drain time, validation, restart, compilation, and data staging are inside the clock.

Record monotonic timestamps for step start and end, optimizer commit, checkpoint staging start, durable checkpoint commit, validation start and end, and any blocking wait. Also record useful units per step. For language models, valid non-padding tokens are often more informative than sequences because padding composition changes. For image training, unique images processed may be the natural unit. Keep the definition stable across systems.

GPU event timing and kernel summaries do not replace application wall time. NVIDIA’s Nsight documentation distinguishes kernel activity from the full application timeline and warns that profiling tools can add overhead or alter execution. Use a monotonic host clock for the headline sustained rate, and use profiler traces for attribution on a short representative capture. Label profiled throughput separately if instrumentation changes it.

Consider an illustrative 20-minute window. Training processes 9,600,000 valid tokens during 900 seconds of active steps, so active-step throughput is 9,600,000 / 900 = 10,667 tokens per second. Two synchronous checkpoints consume 90 seconds total and one validation consumes 210 seconds. Wall time is 900 + 90 + 210 = 1,200 seconds, so sustained throughput is 9,600,000 / 1,200 = 8,000 tokens per second.

The difference is material: sustained throughput is 8,000 / 10,667 = 75 percent of the active rate, with rounding. These figures are illustrative and do not predict any model or storage system. Report the work total, each duration component, both formulas, and the cadence. A reader can then evaluate a faster checkpoint design without confusing it with faster training kernels.

An asynchronous checkpoint may reduce the visible pause while consuming host memory, storage bandwidth, CPU time, and network bandwidth concurrently with training. Measure step throughput before, during, and after staging, plus time until the durable commit. If training finishes before the upload drains, include the final drain when the recovery contract requires that checkpoint to exist. Calling the save free because optimizer steps continued is an accounting error.

Track queue depth and checkpoint age. If a new snapshot arrives faster than the previous one can become durable, backlog grows until memory, disk, or retention limits force a stall. A measurable rule could require the 95th-percentile durable commit latency to remain below the checkpoint interval and active throughput during staging to stay within 5 percent of the non-staging median. The 5 percent threshold is illustrative and should be set from the service objective.

Start with the identity sustained rate equals useful work divided by total wall time. Partition wall time into active training, synchronous checkpoint blocking, validation, data waits, scheduler or barrier waits, and unexplained time. Then inspect overlap: asynchronous checkpoint work may appear inside active training but depress step rate. Compare step-time distributions inside and outside each annotated state before attributing the loss to storage.

Run controlled changes one at a time. First disable checkpointing for a short diagnostic, not for the headline result. Next keep checkpoint frequency but direct output to a known destination or vary serialization concurrency. Then move validation without changing its sample count. Use system traces to locate CPU, CUDA, and I/O overlap after application accounting identifies the suspect window. This sequence prevents a kernel metric from being blamed for time spent at a barrier.

A useful scorecard includes model and precision, global batch or valid tokens per update, GPU topology, dataset and sequence-length distribution, window length, useful work, active-step rate, sustained rate, checkpoint size and cadence, durable commit latency, validation cadence, and every exclusion. Include median and tail step times across multiple windows rather than selecting the quietest interval.

Adopt a capacity change only if sustained throughput improves beyond normal run-to-run variation while recovery and quality checks remain unchanged. An illustrative decision rule is at least three comparable windows, no checkpoint backlog growth, and a sustained-rate gain above 3 percent with unchanged valid-token accounting. The threshold is illustrative. ClusterBid capacity comparisons should use this sustained workload contract; active-step speed alone can overstate delivered progress when storage and validation occupy a recurring share of the run.

MLCommons Training Rules

NVIDIA Nsight Systems User Guide

NVIDIA Nsight Systems Analysis Guide

vLLM vs SGLang vs TensorRT-LLM 2026 Inference Showdown: Throughput, Latency, Cost

A10 24GB vs A100 80GB GPU Comparison 2026: Entry Ampere vs Flagship - Performance, Price and Best Workloads