Training Tokens per Second Need a Sequence-Length Footnote

Training tokens per second looks like a complete rate, but the numerator can mean scheduled token slots, non-padding input tokens, or tokens that actually contribute to the loss. Those quantities diverge as soon as examples have uneven lengths. Within a run that schedules 4,096 positions per sequence but averages 2,048 valid positions, scheduled-token throughput is twice valid-input-token throughput if both use the same elapsed-time denominator. The sequence-length footnote is therefore part of the measurement, not editorial detail.

Publish at least two rates when padding exists: scheduled-token throughput and valid-token throughput. Scheduled tokens per step equal global sequences per step multiplied by the fixed sequence length. Valid tokens require summing the attention mask or the loss mask over the measured steps. If some tokens attend but are excluded from loss, add loss-bearing-token throughput. The reader can then distinguish tensor work from training signal without guessing which denominator produced the headline.

A comparable run fixes model revision, tokenizer, dataset slice, example-order seed, precision mode, optimizer, parallelism layout, sequence-length policy, and global batch definition. It also records whether compilation, data loading, validation, checkpoint writes, or logging are inside the timed boundary. NVIDIA's Megatron-LM benchmark notes that its end-to-end figures include data loading, optimizer steps, communication, and logging. That kind of boundary statement is more useful than a bare rate.

Precision belongs in the headline metadata because it can change kernels, memory traffic, numerical behavior, and feasible microbatch size. Keep loss scaling and optimizer-state precision explicit as well. Warm-up should be excluded by a declared rule, such as discarding a fixed number of steps and reporting the median plus a tail percentile over the remaining contiguous window. Do not choose the window after looking for the fastest interval.

Padding fixes tensor shapes but spends attention and feed-forward work on positions that may not carry data. Packing several shorter examples into one sequence can raise the valid-token fraction, yet it also changes the batch composition and may introduce segment masks, document-boundary handling, or different kernels. A packed result and a padded result answer different operational questions unless every other condition is held constant and both scheduled and valid token counts are shown.

Report the length distribution, not only the maximum. At minimum include median, p95, maximum, and the valid-token fraction for the measured window. If batches are bucketed by length, describe the bucket rule and whether samples were reordered. If a framework drops or carries a remainder batch, state that too. These details reveal whether a speedup came from executing the same workload better or from feeding the accelerator a different workload.

Consider an illustrative eight-GPU run with global batch 256 sequences, fixed length 2,048, and median step time 1.60 seconds. Scheduled throughput is 256 x 2,048 / 1.60 = 327,680 token slots per second. Suppose the summed loss mask contains 314,573 valid tokens per step. Valid-token throughput is 314,573 / 1.60 = 196,608.125, or approximately 196,608 valid tokens per second. The valid-token fraction is 314,573 / 524,288 = 60.0 percent after rounding. Every number is illustrative, not a measured hardware claim.

Now suppose packing raises the valid-token count to 471,859 but step time becomes 1.76 seconds. Scheduled throughput is approximately 297,891 token slots per second, while valid-token throughput is approximately 268,102. The opposite movement is the point: ranking by scheduled tokens would reject the packed configuration, while ranking by useful training tokens would accept it. Preserve the same dataset slice and optimizer-step workload before attributing the difference to packing alone.

For data parallel training, describe global batch in sequences and in valid tokens. The familiar sequence count can conceal a large token-count shift when length distributions or packing policies differ. Also state microbatch size, data-parallel replica count, and accumulation count. A throughput comparison at global batch 1,024 against one at 256 changes kernel shapes, synchronization frequency, and the number of optimizer updates per token budget.

A useful comparison either holds global valid tokens per optimizer step within a declared tolerance or labels the run as a scaling study rather than a drop-in performance contest. Keep learning-rate schedule coordinates consistent: steps, samples, and tokens are not interchangeable. A faster run that performs fewer optimizer updates over the same token budget may be operationally attractive, but it is not evidence of equal convergence.

Adopt a configuration only if valid-token throughput improves by at least 5 percent in three repeated runs, the median loss over a short correctness window remains within a predeclared tolerance, and peak allocated memory stays below an operating ceiling such as 90 percent of device capacity. The 5 percent and 90 percent thresholds are illustrative policy choices, not universal constants. Record dispersion so a noisy 5 percent difference is not mistaken for a stable gain.

Diagnose a surprising result in order. First verify mask sums and tokenizer output. Second compare sequence-length histograms and valid fractions. Third confirm global batch, accumulation, precision, and parallel layout. Fourth inspect step-time components and synchronization. Fifth repeat with synthetic or cached input to bound loader influence. Only then inspect kernels. This sequence catches denominator and workload errors before expensive low-level profiling.

Even a disciplined throughput card does not establish time to convergence, final quality, or resilience under long runs. Packing can alter sample adjacency; different sequence lengths can change the optimization problem; and short timing windows can miss checkpoint pauses, allocator fragmentation, thermal behavior, or storage contention. Treat the rate as a capacity input bounded to the stated workload.

When matching capacity through ClusterBid, carry the entire rate card with the requirement: model and software revisions, GPU count, precision, parallel layout, global batch, sequence statistics, valid fraction, and timed boundary. A provider result is comparable only when those fields align closely enough for the intended decision. The honest output is sometimes a range, not a single tokens-per-second number.

NVIDIA Megatron-LM repository and benchmark configuration

Megatron-LM training arguments source

PyTorch DistributedDataParallel tutorial

vLLM vs SGLang vs TensorRT-LLM 2026 Inference Showdown: Throughput, Latency, Cost

A10 24GB vs A100 80GB GPU Comparison 2026: Entry Ampere vs Flagship - Performance, Price and Best Workloads