Why Single-Pair Network Tests Overestimate Cluster Throughput

A bandwidth test between two idle hosts answers a narrow and useful question: what did that flow achieve over that path during that interval? It does not show how many other flows share the same uplink, whether a collective crosses rack boundaries, or how endpoint PCIe and memory paths behave when every GPU communicates. Treat it as a component check, not a capacity forecast.

The gap grows with cluster size. Two hosts may remain under one leaf switch, while a 64-node job crosses several leaves and spine links. One stream may occupy most of a path that dozens of simultaneous flows must share. Even a line-rate pair result is compatible with much lower per-node throughput under fan-in, incast, queue contention, or an oversubscribed fabric.

All-reduce is not one transfer between a chosen sender and receiver. Every rank contributes data and receives the reduced result through an algorithm such as a ring or tree. All-gather and reduce-scatter impose different byte movement and scheduling. NCCL may select algorithms and channels according to topology and message size, so the traffic pattern can change across a sweep even when the same hosts participate.

This is why raw network gigabits per second should not be compared directly with a collective result. nccl-tests reports algorithm bandwidth and a derived bus bandwidth whose conversion depends on the collective and rank count. Use its published formulas when interpreting results, and keep operation, ranks, datatype, message size, and in-place mode attached to every number.

Suppose eight nodes each have a nominal 100 Gb/s host link but share one 200 Gb/s uplink in the tested direction. A pair within the rack may approach the host-link envelope. If all eight send across that uplink at once and divide it equally, the aggregate directional constraint gives a simple ceiling of 200 / 8 = 25 Gb/s per sender before protocol overhead. This is illustrative arithmetic, not a measured fabric result; actual hashing, receiver placement, full-duplex traffic, and collective schedules vary.

Shared constraints also exist inside servers. Multiple GPUs may traverse one NIC, PCIe switch, CPU root complex, or NUMA link. A CPU-memory pair test can bypass the GPU-memory path, while one-GPU testing can leave other rails idle. Capture GPU-to-NIC affinity and run with the production number of GPUs per host so the endpoint is loaded in the same shape as the intended job.

Start with a matrix of representative pairs: within a node if applicable, within a leaf group, across leaf groups, and across every rail. Sweep message sizes and both directions. Next run several disjoint pairs simultaneously, then a many-to-one fan-in test if storage or parameter traffic has that shape, and finally full-rank collectives. Preserve interface counters and per-flow results rather than only the aggregate.

At each stage, hold image, CPU affinity, NUMA placement, MTU, and test duration constant. If throughput falls when the fourth pair starts, repeat with different host combinations to see whether the boundary follows a rack or endpoint. Do not tune switch queues from tenant nodes. Ask the provider to correlate the timestamp with port utilization, drops, ECN marks, or PFC events where those apply.

A training job sends more than one large gradient buffer. It can combine small control operations, medium and large collectives, checkpoint traffic, data reads, telemetry, and rendezvous messages. Export a message-size histogram or framework communication trace from a safe representative run. Recreate the major sizes and operations, including burst timing, instead of benchmarking only the size that produces the largest headline bandwidth.

Then run a short application window with the same global batch, parallelism strategy, sequence-length distribution, and storage behavior. Measure tokens or samples per second, median and p95 step time, collective time share, and per-rank stragglers. A synthetic collective can validate the fabric yet still miss synchronization caused by data loading or compute imbalance. Conversely, a slow application does not prove the network is slow.

Choose a consistent per-node workload shape and define aggregate throughput T(n) at n nodes. One useful scaling metric is E(n) = T(n) / (n x T(1)). If one node processes an illustrative 1,000 samples per second and eight nodes process 6,400 with the same per-node batch and other relevant settings, then E(8) = 6,400 / 8,000 = 0.80, or 80 percent. This measures throughput scaling efficiency under that setup; it does not isolate network cost or guarantee equivalent optimization behavior. The numbers are illustrative and actual results vary.

Set a rule before procurement, such as median E(n) of at least 85 percent at the target node count, p95 step time no more than 15 percent above the modeled target, and zero correctness failures across three runs. Also require that the slowest rank's collective time remain within a chosen spread of the median. ClusterBid can then match capacity to the tested node count and topology requirement rather than to a single-pair link label.

Report median, p95, minimum, and maximum by message size and rank, plus every failed or retried operation. Do not average away a single slow path because synchronous training progresses at the pace of constrained participants. Include warm-up policy and raw iteration counts. Compare like with like, especially when nccl-tests rank count or GPU-per-thread settings change.

The limitation is temporal coverage. A clean one-hour acceptance test cannot exclude congestion during another tenant mix or after a topology change. Retain the recipe as a canary and rerun after node replacement or maintenance. If pair tests stay healthy while the full collective regresses, escalate with the topology map, concurrent results, and timestamps. That contrast is precisely the evidence a single-pair benchmark cannot provide.

NVIDIA nccl-tests README

Performance reported by NCCL tests

NCCL Environment Variables

vLLM vs SGLang vs TensorRT-LLM 2026 Inference Showdown: Throughput, Latency, Cost

A10 24GB vs A100 80GB GPU Comparison 2026: Entry Ampere vs Flagship - Performance, Price and Best Workloads