Screen the node before the job builds a world
One slow node can hold every rank at a synchronization point, but a node should not be quarantined because one training step looked long. Queue delay, cold caches, data imbalance, compilation, and a busy shared service can all create a transient straggler. Admission testing needs a short, controlled profile that runs while the candidate is reserved and idle, before it joins a large allocation.
Build baselines per approved hardware and software class. Nodes with different GPU models, CPU generations, memory population, local-drive layouts, NICs, firmware, or power policies should not share a single threshold. Record GPU UUIDs and stable node identity so replacement hardware is not mistaken for historical continuity. Use the same container digest, test inputs, affinity, and sampling intervals on every candidate. Run the check from a scheduler prologue or a dedicated admission queue so an ordinary job cannot overlap it. Record whether caches are cold or warm, since an undocumented cache difference can look like a storage defect.
Measure four independent paths
The compute check should sustain a known operation long enough to expose unstable clocks or thermals, while the memory check verifies allocation, integrity, and local bandwidth. The storage check should use a fixed file size and access pattern against the path the training job will use, distinguishing local scratch from the shared dataset service. The fabric check should measure GPU-to-GPU paths inside the node and the node-to-node path through the intended NIC and switch domain.
NVIDIA DCGM documents diagnostics for compute, memory, memory bandwidth, PCIe, stress, and GPU copy paths. NVIDIA's nvbandwidth measures transfer bandwidth across different links and verifies copied data unless verification is explicitly skipped. These are useful building blocks, not interchangeable scores. A passing memory test does not prove storage health, and a fast device copy does not prove that an all-reduce will select the expected network path.
Use a robust peer-relative baseline with an absolute floor
For each signal, retain a rolling reference from recently accepted, idle nodes of the exact class. Prefer the median and a robust spread measure over the mean when a few sick nodes could contaminate the population. Add an absolute floor derived from workload needs. A peer-relative rule alone can accept an entire pool that has degraded together, while an absolute rule alone can miss one node that is materially slower than otherwise identical peers.
A practical gate is two-stage: flag a candidate if any metric is below the larger of the absolute floor and a stated fraction of the peer median, then run one confirmation after checking exclusivity and environment parity. Quarantine only if the initial run and confirmation fail in the same domain, or if one run produces a reproducible data-integrity failure or a documented fatal hardware event. Keep warning and quarantine separate. A marginal result can shorten the retest interval without immediately removing capacity. Expire old reference samples after planned software or firmware changes. Otherwise the gate may compare a newly tuned pool with measurements from a configuration that no longer exists.
Worked example: decide whether node 27 enters the allocation
Assume an illustrative class baseline from 20 healthy nodes: compute proxy median 980 units/s, GPU memory bandwidth proxy 1,500 GB/s, local read throughput 6.0 GB/s, and large-message collective bus bandwidth 300 GB/s. Node 27 reports 968 units/s, 1,472 GB/s, 5.9 GB/s, and 258 GB/s. These values are invented only to show the calculation and are not hardware specifications or expected performance.
With a rule that flags results below 90 percent of the class median, the fabric floor is 300 x 0.90 = 270 GB/s. Node 27 reaches 258 / 300 x 100 = 86 percent, while its other domains exceed 90 percent. Drain background work, verify rank and NIC placement, and repeat the fabric test once under the same controlled conditions. If that confirmation remains below 270 GB/s, quarantine the node from distributed training but preserve access for diagnosis. If it recovers and no errors appear, admit it with a warning and a scheduled recheck.
| Domain | Controlled input | Quarantine evidence |
|---|---|---|
| Compute | Fixed operation, duration, affinity, and power policy | Reproducible low rate or computation error |
| GPU memory | Fixed buffer and verified access pattern | Integrity failure or repeated bandwidth breach |
| Storage | Fixed file, cache state, queue depth, and mount | Repeated latency or throughput breach on intended path |
| Fabric | Fixed peers, ranks, sizes, NIC, and switch domain | Correctness error or repeated bandwidth breach |
Follow the failing domain instead of rerunning everything
For a compute failure, inspect active processes, application clocks, power draw, temperature, clock-throttle reasons, ECC or Xid events, and CPU affinity. For memory, retain the diagnostic error code and test identity. For storage, compare device latency, filesystem, mount, cache state, and shared-service behavior. For fabric, compare topology, link state, error counters, selected interfaces, peer access, rank binding, and the same test against a known-good peer.
Change one variable per paired test. Move the suspect node to a known-good peer, then keep the peer fixed while changing the network path through approved operational procedures. Compare the suspect GPU pair with another pair in the same chassis. A DCGM failure can originate in host software, permissions, topology, cooling, or interfering work as well as hardware, so its own documentation cautions against treating the tool as an automatic repair or replacement decision.
Keep quarantine reversible and auditable
Quarantine should carry the failed metric, raw output, timestamps, software identity, comparison cohort, retries, and a named release condition. Examples of release evidence include passing the same test twice after a cable replacement or completing a longer administrator-led diagnostic after a cooling repair. Do not return a node merely because it rebooted cleanly. Rebooting changes state, but it does not reproduce the condition that triggered isolation.
Short admission checks are samples. They can miss failures that appear only after hours, under full-cluster contention, or during checkpoint I/O. They can also reject a healthy node when shared dependencies are noisy. Pair admission checks with production telemetry and periodic deeper tests. Track false rejects and later job stragglers by node class, then adjust thresholds through a reviewed change rather than ad hoc exceptions. The operational win is not perfect prediction; it is preventing reproducible stragglers from entering expensive allocations while avoiding unsupported quarantine decisions.