A healthy device is a scoped claim
A green inventory check can confirm that GPUs enumerate, drivers load, memory telemetry is readable, and temperatures look normal. It cannot prove that every GPU pair exchanges data correctly, every NVLink is active, the PCIe path performs as expected, or the network transport works across nodes. NVIDIA's DCGM documentation explicitly separates passive health monitoring from active diagnostics and notes that a node can have no passive incident yet fail an active test.
Collective tests add another layer. They exercise multiple devices, synchronization, topology discovery, libraries, peer access, and possibly NICs and switches. A failure can originate in hardware, firmware, process launch, permissions, topology, interface selection, or a software mismatch. Treat the initial health pass as one completed branch of evidence, not as proof that the failing collective must be an application bug.
Freeze the failure signature before changing anything
Record the exact collective command, host list, rank map, environment, NCCL and CUDA versions, driver and NIC firmware versions, container digest, scheduler allocation, and first error from every rank. Save topology output and the collective's correctness line. Distinguish a hang, explicit transport error, CUDA error, and numerical mismatch. They point to different next tests.
Repeat once with the node reserved and no unrelated jobs. Use a bounded timeout and collect logs from all ranks. Do not begin by forcing algorithms, disabling peer paths, or changing several interfaces at once. Those changes can route around the defect and erase the original signature. The first goal is reproducibility under a known placement.
Reduce the topology to one edge at a time
Test a single GPU locally to verify context creation and basic computation. Then run pairwise copies or a two-rank collective across each relevant GPU pair inside the new node. Follow with all local GPUs. Next, pair one GPU and NIC on the new node with one known-good peer node. Only after those pass should the full intended multi-node rank set return.
Represent the system as edges: GPU to local memory, GPU to GPU, GPU to PCIe root, GPU to NIC, NIC to transceiver, transceiver to switch port, and switch path to the peer. A test should add as few edges as possible. When the smallest failing test crosses one new edge, inspect that edge and both endpoints. A transceiver cage with link up is not cleared by link state alone; optical errors, FEC behavior, lane state, speed, MTU, and traffic counters still matter.
| Test scope | If it passes | If it fails |
|---|---|---|
| One GPU | Basic runtime and device path work | Investigate device, driver, permissions, or local software |
| Two local GPUs | That peer path works | Inspect pair topology, peer access, PCIe, or NVLink |
| All GPUs in one node | Local collective works at full rank count | Inspect local topology or one failing pair |
| One GPU across two nodes | Minimal network path works | Inspect NIC, transceiver, switch path, MTU, routing, or transport |
| Full multi-node group | Intended topology works | Investigate scale, shared paths, rank mapping, or contention |
Use a pairwise map to identify the common endpoint
Consider an illustrative eight-GPU node. Tests involving GPU 6 fail against GPUs 0, 1, 2, 3, 4, 5, and 7, while every pair not containing GPU 6 passes. That is seven failing edges sharing one endpoint, so GPU 6's peer path, its PCIe attachment, or its local link state becomes the leading diagnostic area. This is illustrative logic, not a claim about a particular server.
Now change the pattern: every local pair passes, but any inter-node test using NIC 1 fails while NIC 0 passes to the same known-good peer. The common boundary has moved from the GPU mesh to the NIC 1 path. Swap only rank-to-NIC affinity or the external path as a controlled test, then inspect counters on both ends. A decision rule is to quarantine the node when a correctness failure repeats three times on the same minimal edge while equivalent edges pass. Three is an illustrative operational threshold; critical environments may quarantine after one repeatable mismatch.
Active diagnostics fill the gap left by polling
Run supported active diagnostics only on drained resources. DCGM describes tests that exercise memory, compute, PCIe, NVLink, power, thermal behavior, and collective paths. Its documentation also warns that diagnostics consume system capacity and may conflict with running processes. Select the test related to the smallest failing scope rather than assuming the longest suite is always the fastest route to an answer.
Compare diagnostic evidence with the pairwise map. A PCIe or NVLink diagnostic failure that shares the same endpoint strengthens the hardware-path hypothesis. A clean result narrows the field but does not invalidate the collective failure, because the workloads and paths may differ. DCGM also states that its diagnostics do not repair faults or determine return eligibility. Preserve vendor-specific logs before resets or component changes.
Inter-node tests require evidence from both ends
Capture host NIC counters, switch-port counters, link state, negotiated speed, FEC counters, MTU, routing, and transport logs before and after a short failing window. NVIDIA's Cumulus documentation advises checking whether physical error counters are actively increasing rather than relying on accumulated values that may date from a link transition. Compare the new node's port with the known-good peer and with another equivalent port.
If a counter increases, calculate a delta and rate. An illustrative example is 120 new uncorrectable events during a 60-second reproduction, or 2 per second. That temporal match deserves investigation, but it is not proof that those events caused the collective failure. Conversely, a lifetime count of 20,000 with a zero delta during the failure is historical evidence, not a live correlation. Avoid reseating optics or altering FEC while the node is in service; follow the operator's approved maintenance procedure.
Return to service only after layered evidence
A repaired node should pass the minimal test that exposed the fault, the full intra-node collective sweep, a known-good inter-node pair, and the intended multi-node rank layout. Include correctness and performance across relevant message sizes. Compare against equivalent nodes with the same software and topology, using a predefined tolerance rather than an attractive single run.
An illustrative release rule is three consecutive correct runs with median bandwidth within 10 percent of the equivalent-node baseline at workload-critical sizes, plus no new relevant error-counter deltas. The 10 percent and three-run values are policy examples, not vendor limits. The limitation is unavoidable: passing commissioning reduces known risk but cannot simulate every concurrent route or application phase. Keep the node identity in early production telemetry so a recurring signature remains traceable.