Define the purchasing or operations question first
An afternoon fabric benchmark should answer a bounded question: are the allocated GPU paths consistent enough for a target distributed workload, and where does performance change as traffic leaves a node? It is not a certification of every link, switch route, collective, or failure mode. A small matrix becomes useful when every row corresponds to a decision, such as accepting a node pair, escalating an outlier, or choosing a parallel layout.
Write the primary metric and stopping rule before testing. For large transfers, that may be sustained bus bandwidth relative to a previously accepted same-class baseline. For small transfers, it may be median and tail latency. For collective behavior, include both correctness and spread across repetitions. Avoid using a vendor line-rate number as the only denominator because protocol, topology, collective algorithm, and endpoint limits affect observed application throughput.
Choose four axes and resist expanding them
Use four axes: operation, message size, path class, and concurrency. A compact operation set is point-to-point RDMA write bandwidth plus NCCL all-reduce. Add all-gather only when the target workload materially depends on it. Select three message sizes from actual traces if available: one small enough to expose latency, one near a common bucket or activation size, and one large enough to approach steady bandwidth.
Path classes matter more than testing every random pair. Choose one pair inside a fast local GPU domain, one pair across local PCIe or socket boundaries if that path exists, one adjacent-node network pair, and one pair that crosses the suspected shared uplink. Repeat an outlier on a second pair of the same class before labeling the whole class. This keeps coverage interpretable without turning the exercise into a topology census.
| Matrix axis | Compact choice | Question answered |
|---|---|---|
| Operation | RDMA write, NCCL all-reduce | Is the link healthy, and can the collective use it? |
| Size | Small, workload-sized, large | Where do latency and bandwidth regimes change? |
| Path | Local fast, local slow, network near, network far | Which boundary creates the drop? |
| Concurrency | One flow, representative parallel flows | Does sharing collapse useful throughput? |
Budget the runs before opening the matrix
An illustrative full factorial has 2 operations x 3 sizes x 4 path classes x 2 concurrency levels = 48 configurations. Three repetitions would produce at most 144 measured runs. At 30 seconds of measurement plus 30 seconds for launch, capture, and reset per run, that is 144 x 60 seconds = 8,640 seconds, or 2.4 hours. Reserve another hour for inventory, warmup, reruns, and packaging. These timings are illustrative; actual launch overhead and required duration vary.
Do not mechanically run every combination when a tool makes a cell meaningless. RDMA perftest and NCCL tests exercise different layers and accept different parameters. Keep their results separate. NCCL tests can sweep byte ranges and reports collective-oriented bandwidth fields. The linux-rdma perftest tools include duration mode, queue-pair count, JSON output, counters, and separate bandwidth or latency programs. Record the exact command rather than normalizing unlike outputs into one score.
Control the test window and randomize obvious bias
Warm each path before recording it, then use a fixed measurement duration. Ensure no checkpoint, data copy, health scan, or unrelated training job is intentionally scheduled on the test nodes. For a shared environment, note that absence of visible local work does not prove absence of network contention. Capture start and end timestamps so switch or provider telemetry can be aligned later.
Alternate path classes instead of running all best-looking paths first and all remote paths last. A simple rotation helps distinguish topology from a time-varying background load. Run the concurrency test both before and after the single-flow sweep when time permits. If results drift in one direction, the benchmark window itself may be nonstationary and a single median can conceal that fact.
Archive enough configuration to reproduce a surprise
Save hostnames, timestamps, GPU model and PCI address, NIC device and port, link state, negotiated width or speed where exposed, CPU and NUMA topology, driver, firmware, CUDA, NCCL, MPI, benchmark commit or package version, container digest, environment variables, launcher host order, and commands. Preserve raw stdout and stderr before producing a summary table. A screenshot is not a raw result.
NCCL relies on sysfs for PCI topology and can select interfaces automatically, so retain its initialization log and topology dump when diagnosing path choice. For RDMA, save GID, MTU, queue-pair count, directionality, and whether GPU memory or host memory backed the test. Configuration drift can otherwise create a plausible but false before-and-after story.
Apply a measurable gate without inventing universality
A practical gate compares like with like. For each path class and size, compute the median of three repetitions and the range. An illustrative policy flags a cell if its median is more than 10 percent below the accepted same-hardware baseline or if its slowest repetition is more than 15 percent below its fastest. Those percentages are operational choices, not standards, and should reflect normal variation observed in your environment.
For example, three illustrative large-message results of 182, 180, and 151 GB/s have a median of 180 GB/s. The slowest-to-fastest drop is (182 - 151) / 182 x 100 = 17.0 percent, so the cell fails the illustrative 15 percent variability rule even though its median may look strong. Actual throughput differs by topology, versions, collective, and load. Rerun that path while capturing counters before changing tuning.
Interpret the matrix as a locator, not a verdict
If raw RDMA bandwidth is weak and the collective is weak on the same node pair, investigate the network path, NIC placement, and counters. If RDMA is stable but all-reduce is not, inspect rank mapping, GPU-to-NIC locality, NCCL interface selection, and collective configuration. If only concurrent tests degrade, shared-port or fabric contention becomes a diagnostic possibility, not a proven root cause.
This matrix will miss rare thermal behavior, long-duration error accumulation, alternate equal-cost routes, and workload-specific overlap. It also cannot prove isolation in a multi-tenant network. Use it to select the next deeper test and to match measured requirements to candidate capacity through ClusterBid. Preserve the evidence bundle so a later software or cabling change can be compared against the same procedure.