NCCL All-Reduce Results: Bus Bandwidth and Algorithm Bandwidth

An NCCL test row usually invites a quick comparison of the largest number, but time, algorithm bandwidth, and bus bandwidth answer different questions. Time is the reported average duration per collective under the test settings. Algorithm bandwidth, commonly labeled algbw, is payload size divided by duration. Bus bandwidth, or busbw, applies a collective-specific factor so the result can be compared more sensibly with the communication capacity exercised by the operation. None of these values alone is an application throughput forecast.

For small messages, elapsed time is normally the most informative field because fixed work such as kernel launch, synchronization, and protocol progress is large relative to data movement. For large messages, the slope of time against bytes and the bandwidth plateau are more useful. A report should therefore preserve the whole size sweep, not just the peak busbw row. It should also record NCCL, driver, firmware, GPU topology, rank placement, process layout, data type, reduction operator, and whether the test used in-place or out-of-place buffers.

NVIDIA's nccl-tests documentation defines all-reduce bus bandwidth as algbw multiplied by 2 times the quantity ranks minus one, divided by ranks. The factor represents the reduction and distribution traffic of a conventional point-to-point collective model. With eight ranks, the factor is 2 x 7 / 8, or 1.75. With two ranks it is 1.0. This is why busbw can be larger than algbw without indicating that bytes appeared from nowhere.

Consider an illustrative row with a 256 MiB message and a 10.0 ms time across eight ranks. Algorithm bandwidth is 268,435,456 bytes / 0.010 seconds / 1,000,000,000, which is 26.84 GB/s. Bus bandwidth is 26.84 x 1.75, which is 46.97 GB/s. These figures are illustrative arithmetic, not a measured platform result. Changing to 16 ranks changes the normalization factor to 2 x 15 / 16, or 1.875, even if elapsed time were unchanged. Always recompute or verify the factor before comparing exported results.

A logarithmic sweep from bytes through MiB exposes at least three regimes. At the left, fixed overhead dominates and bandwidth calculated from a tiny numerator is necessarily low. In the transition region, additional channels, protocols, or algorithms may become favorable. At the right, transfer time is closer to overhead plus bytes divided by sustained bandwidth. The exact knees are platform and software dependent, so copying a threshold from another cluster is not a sound acceptance rule.

Use the size distribution of the intended workload to weight the curve. If gradient buckets are mostly 16 MiB, a peak at 1 GiB is weak evidence. If control collectives occur thousands of times per step at a few KiB, report microseconds per operation and their frequency. A practical decision rule is to require the candidate system to meet the agreed latency ceiling at the workload's 50th and 95th percentile message sizes, and meet the agreed bandwidth floor over its large-message range. Define those percentiles and floors before offers are compared.

Rank count affects more than the formula. One GPU pair may stay on a direct NVLink path, eight ranks may traverse an NVSwitch or PCIe hierarchy, and ranks crossing nodes add network adapters and switches. A 16-rank result is therefore not an eight-rank result multiplied by two. Placement matters too: eight ranks packed into one node answer an intra-node question, while eight ranks distributed across eight nodes answer an inter-node question.

Build a rank matrix rather than a single scale-up line. At minimum, test two GPUs within the same locality, all GPUs in one node, two nodes at full local GPU occupancy, and the intended multi-node job size. Keep message sizes and test options constant. If performance drops, the first rank count at which it drops identifies a topology boundary to inspect. Do not label that boundary faulty until repeated placement tests distinguish a shared path constraint from a single node, adapter, or link.

The nccl-tests performance note gives different factors because collectives move and concentrate data differently. All-reduce uses 2 x (n - 1) / n. Reduce-scatter and all-gather use (n - 1) / n. Broadcast and reduce use 1. An all-gather busbw value and an all-reduce busbw value are therefore products of different normalizations, even when their payload size labels match.

Compare like with like: same collective, rank count, placement, message-size definition, and software configuration. For reduce-scatter and all-gather, verify whether the displayed size represents the total array or the per-rank count in the invocation. When a training framework decomposes a logical all-reduce into reduce-scatter plus all-gather, profile those actual calls rather than assuming a standalone all-reduce curve predicts them exactly. The honest limitation is that busbw is a model-derived normalization, not a physical counter for one particular NVLink or network port.

First, archive the exact command and environment. Second, confirm correctness output before reading performance. Third, plot time and both bandwidth fields against size. Fourth, repeat at topology boundaries and swap rank placement. Fifth, compare medians across repeated runs and retain dispersion rather than choosing the best pass. Sixth, map workload message sizes and collective counts onto the relevant part of each curve. This order prevents a fast but incorrect run, or a favorable placement accident, from becoming the purchasing baseline.

A measurable escalation rule can be simple: investigate when the median time at a workload-critical size is more than 15 percent slower than the approved baseline in three consecutive isolated runs, or when one placement is more than 15 percent behind equivalent placements. The 15 percent value is an illustrative policy threshold, not a universal NCCL tolerance. Set it from normal variation in the buyer's environment. Match capacity offers to the measured rank layout and bandwidth floor rather than to an unqualified peak number.

A clean nccl-tests curve demonstrates correctness and performance for its buffers, collective, process arrangement, and measurement interval. It does not prove that a model will overlap communication with compute, that the data loader will keep GPUs busy, or that shared network traffic will remain absent. It also does not isolate which physical link carried every byte. Hierarchical algorithms can use intra-node and inter-node resources differently from the simple normalization model.

Keep the result as a reproducible component test. Pair it with application traces, interface counters, topology records, and concurrent-load tests when those conditions matter. If two suppliers show only peak busbw, request raw per-size output and the rank map. The defensible comparison is not whose single number is larger, but whose measured curve satisfies the application's sizes, synchronization frequency, placement, and concurrency assumptions.

NVIDIA nccl-tests: Performance reported by NCCL tests

NVIDIA nccl-tests README

vLLM vs SGLang vs TensorRT-LLM 2026 Inference Showdown: Throughput, Latency, Cost

A10 24GB vs A100 80GB GPU Comparison 2026: Entry Ampere vs Flagship - Performance, Price and Best Workloads