The same model name does not mean the same test system
Two nodes sold under one model name can differ in CPU stepping, DIMM placement, firmware, PCIe negotiation, GPU ordering, NIC locality, cooling position, power policy, kernel, driver, container, or background activity. Even genuinely identical hardware can occupy different thermal and scheduler states. Begin with the narrower claim that node B is slower for one precisely defined workload, not that its GPUs are defective.
Freeze the observation before investigating. Record throughput over a fixed measured interval after a fixed warmup, along with batch and input shape, precision, compilation state, data source, CPU and GPU affinity, and synchronization points. Save per-interval values, not only the average. Record errors, retries, and completed work too, because a higher submitted rate can hide lower useful throughput. Variance within a node and a stable offset between nodes lead to different diagnostic paths.
Use simultaneous paired tests with role reversal
Reserve one fast node and one slow node exclusively. Run the same immutable container and input in parallel so both see the same time window, using separate external capacity or a workload known not to contend for a shared bottleneck. Also run a staggered control; if the gap disappears when the nodes do not overlap, the paired test itself created contention. Then swap external roles such as network peer, dataset shard, switch port where normal operations allow, and rank position. Repeat at least three pairs. If the gap follows the physical node, inspect its local state. If it follows a peer, shard, or rank, the original node label was a distraction.
Pairing controls for cluster-wide load better than comparing a run from Monday with one from Friday. Still randomize which node starts first and alternate test order, because cache warming and compilation can favor the second run. Report the paired ratio for every round. A node that is 4 percent slower in each pair is different from one that alternates between 10 percent faster and 15 percent slower. If the application has variable input lengths, replay the same ordered trace on both nodes instead of relying on two samples from the same distribution.
Inspect clocks, power, and temperature during the measured window
NVIDIA System Management Interface exposes current clocks, power readings, temperature, performance state, and throttle information where supported. Sample during the workload, because idle snapshots can look identical. Compare achieved clocks rather than configured limits alone. A lower clock is an observation; temperature, power limit, synchronization, or another documented throttle reason may be the cause. Sensor availability varies by product, so treat unsupported fields as missing evidence rather than zero.
Correlate telemetry with throughput timestamps. If node B slows only after ten minutes while temperature rises and clocks fall, test with the same workload duration and inspect cooling through approved facility procedures. If clocks remain aligned while throughput differs from the first measured interval, move to topology and software. Do not change power or thermal settings during diagnosis, since that changes both the system and the evidence.
Compare topology, link state, and competing work
Capture nvidia-smi topology matrices and NVLink status from both nodes. Compare GPU-to-GPU paths, GPU-to-NIC locality, CPU affinity, PCIe generation and width where exposed, and active-link state. NCCL documentation notes that topology detection influences communication paths, while nvidia-smi reports whether NVLinks are active. A different logical GPU order can change which devices a fixed rank map pairs even when the chassis wiring is identical.
Next enumerate GPU processes, CPU load, memory pressure, interrupts, storage activity, network traffic, monitoring agents, and container limits. A process consuming a small fraction of GPU time can disturb latency-sensitive synchronization. Check CPU frequency policy and NUMA placement as well, especially when preprocessing or launch overhead is material. Stop only work that operations policy permits, preserve the before-state, and rerun the pair. If removing background work closes the gap, restore components individually to identify the interference rather than declaring the node repaired.
Worked example: locate a reproducible 7 percent gap
Suppose a hypothetical inference proxy produces 10,200 completed requests per minute on node A and 9,480 on node B over equal measured intervals. The ratio is 9,480 / 10,200 = 0.9294, so B is 7.06 percent lower, rounded to 7.1 percent. Three simultaneous pairs show gaps of 6.8, 7.4, and 7.0 percent, and staggered controls preserve the gap. Both nodes hold similar GPU clocks and temperatures, but B's GPU-to-NIC topology differs for the selected devices because the rank launcher uses ordinal order after a device replacement. These values are not product benchmarks.
Reverse the GPU selection so each process uses devices local to the intended NIC, without changing batch or software. The next three paired gaps are 0.9, 1.2, and 0.7 percent. Under an illustrative decision rule, a node is considered divergent only when its median paired gap exceeds 3 percent across three runs and follows the node through a role swap. The corrected median is 0.9 percent, so B returns to service; the durable fix is an explicit UUID-to-NIC placement map, not a looser threshold.
| Observation | Next controlled comparison | Interpretation if the gap follows |
|---|---|---|
| Clock or thermal divergence | Same load and duration, synchronized telemetry | Local power or cooling state |
| Different topology or link state | Explicit device and NIC placement | Rank-to-device path |
| Background activity | Exclusive run, then restore one service at a time | Competing process or host service |
| Software mismatch | Same image on both nodes, then swap one layer | Driver, library, kernel, or configuration |
Bisect the software stack after physical state is controlled
Compare image digest, driver, CUDA-visible runtime, collective library, framework build, kernel, firmware, environment variables, and generated caches. Swap one supported layer at a time. The same application image on both nodes is especially useful, but it does not erase a host-driver or firmware difference. Preserve logs showing the actual libraries loaded and the topology selected, rather than relying on package inventory alone.
A synthetic kernel or transfer test can isolate a subsystem, but it may not reproduce the application's bottleneck. Conversely, an application gap can originate in input handling rather than GPU execution. Retain the test matrix and negative findings, since knowing that clocks, thermals, and topology matched prevents the next investigator from restarting at inventory. The honest stopping rule is evidence-based: close the gap in repeated paired tests or isolate it to a domain with a reproducible minimal test. If neither happens, keep the node in a limited pool and continue collecting evidence instead of assigning a hardware cause from its model name.