One node is the control, not a miniature scale test
Adding nodes multiplies possible causes of a slow or unstable run: network paths, process placement, collective algorithms, storage contention, launch skew, and failures. A one-node acceptance test removes most inter-node variables while exercising the intended model, software image, GPUs, local topology, and input path. Its purpose is to establish a reproducible floor before scale makes attribution expensive.
Define the artifact as a contract: immutable model and data revisions, container or environment digest, driver and library versions, precision, parallel layout, batch construction, seeds, and command line. Save machine-readable results and raw logs. A screenshot of utilization is not a baseline because it cannot be replayed or compared field by field after a change.
Correctness gates performance
Begin with a small deterministic fixture whose samples and expected behavior are known. Confirm finite loss, expected mask counts, gradient presence, parameter updates, checkpoint round trip, and agreement with a trusted reference within declared numerical tolerances. Check every rank for errors. A run that skips updates because of overflow or drops samples can look unusually fast.
Then run a short production-shaped window and record loss, gradient norm, skipped optimizer steps, and samples or valid tokens processed. Exact equality is usually inappropriate across reduced precision and nondeterministic kernels, so specify tolerances before testing. Performance results are invalid if the correctness gate fails; do not average them into the node baseline.
Separate the loader from the training step
Measure three paths: synthetic tensors already on device, cached or memory-resident decoded batches, and the production data pipeline. The first bounds model compute and communication. The second exposes host-to-device movement and collation without storage variability. The third includes file access, decoding, shuffling, workers, and prefetch. The gaps identify where to profile next without claiming that synthetic throughput represents production.
PyTorch's DataLoader supports single-process and multi-process loading, automatic batching, memory pinning, and prefetch controls. Change one control at a time, and watch host CPU, RAM, storage bandwidth, and queue depth along with GPU idle gaps. More workers can increase memory use or storage contention. Accept the loader only when repeated runs feed the step without persistent starvation under the intended production path.
Profile kernels and memory together
After warm-up, capture a bounded profiler trace that contains several complete optimizer steps. Look for large gaps, repeated transfers, unexpected casts, expensive shape changes, and kernels that dominate device time. A profiler perturbs execution, so use it for diagnosis and retain an unprofiled timing run for the acceptance number. Record the precise timed boundary and synchronization method.
For memory, report peak allocated and peak reserved bytes per device, plus the shape that produced the high-water mark. Include parameters, gradients, optimizer state, activations, and framework workspaces in the operational interpretation even if the tool labels them differently. Run the largest supported sequence bucket and leave a predeclared margin. Passing one average batch is not an acceptance test for variable shapes.
Qualify intra-node collectives before model scaling
Run representative collective tests across the same GPUs and process placement the trainer will use. NVIDIA's nccl-tests provides correctness and performance tests, including all-reduce over selectable message sizes. Its documentation distinguishes algorithm bandwidth from bus bandwidth; preserve both along with latency, rank count, topology, and test version rather than copying a single peak figure.
Choose message sizes informed by the model's gradient buckets and parallel collectives, not only a maximum-size transfer. Small messages expose latency while large messages emphasize bandwidth. A passing microbenchmark does not prove the model overlaps communication effectively, but a failing or highly variable microbenchmark is a reason to stop before adding nodes. Compare against an internally qualified node of the same class, not an unrelated marketing number.
A worked acceptance card makes the gate explicit
Consider an illustrative candidate with a frozen 20-step timing window after 10 warm-up steps. The internal reference processes 120,000 valid tokens per second, and policy allows a 5 percent regression, so the floor is 120,000 x 0.95 = 114,000. The candidate median is 116,400, which passes. Its p95 step time must also stay under an illustrative 1.20 times the median to reject unstable nodes.
Suppose the device has an illustrative 80 GiB capacity and policy caps peak allocated memory at 90 percent. The ceiling is 80 x 0.90 = 72 GiB. A measured 69 GiB passes that allocated-memory gate by 3 GiB, but it does not prove 3 GiB is actually free: reserved allocator segments, libraries, the driver, and other processes also consume memory. Check peak reserved memory and process-visible free memory separately. These are hypothetical arithmetic examples, not GPU specifications or observed results. Actual capacity, available memory, workloads, and thresholds vary.
| Illustrative gate | Rule | Candidate | Result |
|---|---|---|---|
| Valid-token rate | >= 114,000/s | 116,400/s | Pass |
| Peak allocated memory | <= 72 GiB | 69 GiB | Pass |
| p95 step time | <= 1.20 x median | Measure required | Pending |
Run the diagnostic ladder in order
If production throughput misses while synthetic passes, inspect the input queue, workers, storage, host-to-device copies, and collation. If both miss, compare kernel traces, power and clock telemetry, precision, compilation state, and process placement. If a single GPU passes but the full node misses, inspect intra-node collectives, topology, rank affinity, and synchronization. If memory alone fails, replay the offending shape and separate allocated from reserved memory.
Change only one factor, rerun the complete gate, and retain the before-and-after artifacts. Do not tune the acceptance threshold to admit a weak node after seeing its result. If a justified software change improves the reference itself, version the baseline and requalify the comparison pool. This turns troubleshooting into elimination rather than speculation.
Issue a scale-readiness receipt
Approve scale-out only when correctness passes, production input keeps the accelerator supplied within the declared tolerance, unprofiled valid-token throughput meets the floor across at least three runs, memory stays below its ceiling for every supported bucket, and intra-node collective results fall inside the internal reference band. Three runs and any numeric bands are policy choices; use more samples when variance is high.
The receipt does not predict multi-node scaling efficiency or final convergence. It says only that the node-level foundation is sound enough to make a multi-node delta meaningful. Carry the baseline when matching measured requirements through ClusterBid, then repeat the same workload at two nodes before a larger allocation. The first scale step should isolate inter-node cost, not rediscover a loader or correctness defect.