Peak bandwidth does not price a barrier
A fabric can move a large buffer quickly and still delay training when the program repeatedly waits on small collectives. The reason is arithmetic: transfer time approaches fixed overhead plus bytes divided by bandwidth. When bytes are few, reducing the second term barely changes the total. Launch, software progress, hop latency, and synchronization determine when every rank may continue.
This matters because a collective is not merely traffic. It is often a dependency in the step graph. A delayed all-reduce for a small gradient bucket, an all-gather for a sharded parameter, or an all-to-all exchange for routing can hold later GPU work. The slowest participating rank determines collective completion, so median link utilization can look comfortable while step time grows.
Count calls before buying more capacity
Start with an application trace that identifies collective type, payload bytes, start time, duration, rank group, and whether useful compute overlaps the call. Summarize call counts and time by logarithmic size buckets such as below 4 KiB, 4 to 64 KiB, 64 KiB to 1 MiB, and above 1 MiB. These boundaries are bookkeeping choices, not universal protocol thresholds. Preserve the raw distribution so a framework change can be compared later.
A realistic profile may mix frequent small synchronization operations, medium gradient buckets, and a few large state transfers. It may also contain bursts: many small all-gathers around layer boundaries followed by compute. Average bytes per step hides that sequence. The decision variable is exposed wait time, meaning collective duration that is on the critical path after overlap, not total communication time alone.
| Profile feature | Question it answers | Common mistake |
|---|---|---|
| Calls per step by size | How often fixed latency is paid | Using average message size |
| Critical-path duration | How much the GPU actually waits | Adding all collective durations |
| Rank group and placement | Which fabric boundary is crossed | Treating every call as cluster-wide |
| Burst timing | Whether queues form around phases | Using only whole-job averages |
Turn microseconds into a step-time budget
Suppose an illustrative training step contains 1,200 small collectives. The measured application trace attributes 18 microseconds of exposed wait to each after overlap. The stall budget is 1,200 x 18 microseconds, or 21.6 ms per step. If the step takes an illustrative 240 ms, these waits account for 21.6 / 240, or 9 percent. The numbers are illustrative, not a benchmark or platform claim.
Now suppose a higher-bandwidth fabric leaves the same small-call latency but halves 8 ms of exposed large-transfer time to 4 ms. The step becomes 236 ms, a reduction of 4 ms, or 1.7 percent of the 240 ms baseline. By contrast, reducing exposed small-call wait from 18 to 14 microseconds saves 1,200 x 4 microseconds, or 4.8 ms. This does not prove latency is always more valuable. It shows why call frequency must be multiplied by exposed duration before choosing the optimization target.
Collective shape changes the latency path
All-reduce, all-gather, reduce-scatter, and all-to-all do not create identical dependency or traffic patterns. An all-reduce returns the reduced value to every rank. All-gather expands shards at every participant. All-to-all creates many peer exchanges and can be sensitive to rank mapping and contention. Control collectives such as barriers may carry almost no payload but make straggler delay visible to everyone.
NCCL can select among algorithms and protocols based on topology and operation characteristics. Its documentation identifies LL, LL128, and Simple protocols and explicitly describes LL buffers for latency-bound point-to-point operations below a threshold. The documentation also discourages forcing protocols except when isolating a suspected bug. Therefore, record selections through supported debug output when diagnosing, but avoid turning one forced setting into a general production recommendation.
Separate fabric latency from a late rank
Use a diagnostic sequence. First, reproduce the call-size distribution with all ranks idle except the test. Second, compare intra-node, adjacent-node, and cross-rack placements at the same rank count. Third, inspect per-rank arrival times around the collective. Fourth, repeat while controlled background traffic is introduced. Fifth, correlate changes with interface queue, drop, error, and pause counters. A collective that starts late on one rank is an upstream compute or scheduling symptom until evidence shows otherwise.
Set a measurable rule around the application, not an abstract microsecond target. For example, require the 95th percentile exposed time from collectives below 64 KiB to stay under 5 percent of median step time during the agreed concurrency test. Both 64 KiB and 5 percent are illustrative policy values. Replace them with boundaries observed in the workload profile. Also require run-to-run dispersion limits, because a low median with a long tail still stalls synchronized ranks.
A fiber path is more than its line rate
The optical path may include GPU-to-NIC traversal, a transceiver, fiber trunks, patch panels, leaf switches, spine links, and the reverse path. Link rate describes serialization capacity at one segment. It does not state end-to-end collective latency, queue residence time, route balance, or whether several racks share uplinks. A clean physical link can still participate in a congested path, while corrected physical errors may exist without causing the observed stall.
Ask a provider for the topology boundary crossed by the proposed rank placement, the number and speed of shared uplinks, and results for small messages under the same concurrent load expected in service. Request per-percentile latency, not only average bandwidth. If the provider cannot reproduce the application mix, supply a compact replay specification with collective types, size histogram, calls per second, rank count, and burst structure.
Treat the model as a budget, not a prediction
Multiplying call count by isolated latency overstates stalls when communication overlaps compute or calls execute concurrently. It can understate stalls when queueing, rank skew, or protocol transitions add tail latency. Application instrumentation is therefore the authority for exposed time. Synthetic tests explain boundaries and support repeatability, but they do not reproduce every framework scheduler decision.
The useful buying record combines two views: large-message bandwidth for sustained transfers and critical-path latency for the observed small-message mix. Keep software versions and rank maps alongside both. Re-run the profile after changing bucket sizes, parallelism strategy, or model architecture, because those choices can alter message frequency more than a faster link changes serialization time.