Reading GPU and NIC Topology Before a Multi-Node Benchmark

A benchmark result belongs to a placement, not merely to a GPU model and node count. Capture every GPU's stable identifier and PCI address, every selected NIC and its PCI address, CPU sockets and NUMA nodes, GPU-to-GPU paths, link state, and the scheduler's visible-device mapping. Save raw command output with the result so a later run can establish whether it used the same hardware path.

NVIDIA's nvidia-smi topology command exposes GPU relationships and CPU or NUMA affinity on supported systems. The hwloc project maps NUMA nodes, processor packages, cores, caches, and I/O devices. Neither view alone is enough: join them by PCI identity. Device index zero can refer to a different physical GPU after a scheduling or visibility change.

For each GPU pair, determine whether traffic uses a direct accelerator link, one or more PCIe switches, a host bridge, or a path that crosses the socket interconnect. For each GPU-to-NIC pair, determine the nearest NIC and whether the route stays under the same PCIe root and NUMA node. The exact labels vary by platform, so retain the tool's legend with the matrix instead of copying unexplained abbreviations into a report.

A short path is a placement hypothesis, not a throughput guarantee. Link generation, negotiated width, switch sharing, firmware, IOMMU configuration, and concurrent traffic can change performance. Record link state and error counters where supported, then verify the path with targeted transfers. Do not alter firmware or low-level settings merely to match a diagram.

Create one row per local rank: host, global rank, local rank, GPU UUID, GPU PCI address, selected network interface, NIC PCI address, CPU set, and NUMA memory node. Generate the table from the launched process environment where possible. A scheduler allocation can be correct while a container renumbers devices or an application binds processes differently.

NCCL says it relies on /sys to discover the PCI topology of GPUs and network cards, and warns that virtual or incomplete PCI topology can lead to suboptimal performance. Verify that the benchmark container sees the expected /sys information. Capture collective-library debug output for interface and transport selection during a diagnostic run, but avoid leaving verbose logging enabled during timed trials unless its overhead has been assessed.

Topology-aware placement depends on what communicates. Tensor-parallel ranks often exchange data frequently and are candidates for placement inside a fast GPU domain. Data-parallel traffic depends on bucketization, accumulation, overlap, and the training framework, so its message size and frequency must come from the trace rather than a rule of thumb. A pipeline stage boundary has its own activation pattern. Write the measured communication edges first, then assign the busiest edges to the shortest measured paths.

For dual-rail nodes, map ranks so each GPU group uses its intended nearby NIC and both rails carry comparable offered load. Avoid assuming that a library will infer the policy you intended. Automatic selection can be good, but the benchmark record should show the chosen interfaces and rank mapping. If two valid mappings exist, test both rather than argue from topology labels alone.

Start with device visibility and PCI identity. Next run GPU-to-GPU pairwise bandwidth and latency within one node. Then test host-to-device transfer with CPU and memory bound to the local and remote NUMA nodes. Test each GPU-to-NIC path with the same message sizes the workload uses. Finally run collective microbenchmarks at one node, two nodes, and the full allocation before starting model code.

At each step, compare symmetric peers rather than an invented universal number. If one GPU pair, NIC path, or rank is materially slower, stop and localize the difference. Inspect negotiated links, interface choice, NUMA binding, errors, and shared-path contention in that order. A model benchmark folds all these paths together and is a poor first diagnostic.

Consider an illustrative node with eight GPUs in two four-GPU link domains. GPUs 0 through 3 are closest to NIC A and CPU NUMA node 0; GPUs 4 through 7 are closest to NIC B and NUMA node 1. The initial launcher assigns local ranks 0 through 3 to the first GPU domain but binds them to NIC B, while ranks 4 through 7 use NIC A. Pairwise network tests show each group crossing the socket boundary. These numbers and relationships are hypothetical.

Remap the first group to NIC A and NUMA node 0, and the second to NIC B and NUMA node 1. The acceptance rule is set before the rerun: every rank must select its intended interface, per-rank collective bandwidth must stay within 10 percent of the node median, and the slower rail must reach at least 90 percent of the faster rail under equal load. Those thresholds are local policy, not vendor specifications. If the gate still fails, inspect fabric-side paths and contention.

Store raw topology output, rank map, CPU binding, library and driver versions, interface configuration, benchmark command, message-size distribution, and timestamps beside the result. Hash a normalized rank-to-resource table and use that hash in comparisons. Reject an apples-to-apples label when the hash changes, unless topology is the variable under test.

Repeat the capture after node repair, firmware or driver maintenance, scheduler changes, container-runtime changes, or movement to another host class. ClusterBid capacity comparisons become more useful when the request specifies GPU count plus measured link-domain, NIC-locality, and NUMA requirements. A product name alone does not encode those conditions.

A topology map can reveal an implausible placement, but it cannot predict every collective. Communication libraries may route around weak paths, use several channels, or select transports based on runtime discovery. Application overlap can hide network time, while synchronization can amplify one slow rank. Keep map validation, microbenchmarks, and end-to-end throughput as three distinct layers of evidence.

The diagnostic also has visibility limits. Virtual machines and containers can expose incomplete topology, and administrative controls may hide counters. In that case, record the missing evidence and avoid a strong locality claim. A credible benchmark report states what the process could see, how placement was verified, and which uncertainties remain.

NVIDIA System Management Interface Documentation

NVIDIA NCCL Documentation: GPU Troubleshooting and Topology Detection

Hardware Locality Project Documentation

GB300 NVL72 Rack-Scale Buyers Guide 2026: Power, Cooling, and Real Cost

Academic GPU Compute Grants: NSF, NIH, and University AI Funding in 2026