Treat the update as a versioned experiment
A driver update changes a layer beneath CUDA applications, collective libraries, peer memory access, and device telemetry. A successful boot and a clean nvidia-smi listing therefore establish only that the driver can enumerate the devices. They do not show that an all-reduce returns correct values on every rank, that NCCL selected the same transport, or that the slow tail of repeated operations stayed within an acceptable range. The useful unit of comparison is a frozen before-and-after experiment, not a one-time smoke test.
Before touching a node pool, record the driver package, kernel, CUDA runtime visible to the test, NCCL and nccl-tests revisions, firmware identifiers exposed by the platform, container digest, command line, environment variables, host list, GPU UUID ordering, and topology output. Preserve the raw output rather than only a dashboard average. A baseline that cannot be replayed with the same binary, message sizes, rank placement, and number of iterations cannot support a defensible rollback decision.
Keep both a functional and a performance baseline
The functional baseline answers whether each collective completes and produces the expected result. NVIDIA's nccl-tests project explicitly checks correctness as well as performance, so capture its error count and exit status for all-reduce plus any collectives the workload actually uses, such as all-gather or reduce-scatter. Test in-place and out-of-place forms when both occur in production. Include small and large buffers because initialization, protocol selection, and sustained transfer exercise different paths.
The performance baseline should retain operation time, algorithm bandwidth, and bus bandwidth by message size. The nccl-tests documentation defines algorithm bandwidth as payload size divided by time. For all-reduce with n ranks, it reports bus bandwidth as algorithm bandwidth multiplied by 2 x (n - 1) / n; other collectives use different factors. Compare like with like. Do not compare an eight-rank baseline to a sixteen-rank candidate or confuse bus bandwidth with application throughput. Store the full size curve because a regression confined to small messages can disappear inside one large-message average.
Stage the test from one node to the real fabric
Start with one unchanged control node and one updated node of the same approved hardware class. On each, capture nvidia-smi topology and NVLink status, then run a single-GPU compute sanity check, pairwise device transfers, and single-node collectives. Only after those pass should the updated node join a two-node test and then the production-size topology. This ordering makes a local peer-access problem distinguishable from a network or multi-node rank-placement problem.
Hold workload variables constant: exclusive access to the GPUs, the same CPU affinity, container image, process launcher, GPU ordering, NIC selection, message-size sweep, warmup count, measured iterations, and power policy. Repeat the candidate run at least three times, interleaving it with control runs if the fabric is shared. Interleaving helps reveal congestion that moves both populations together. A quiet test window is better, but paired controls are still valuable when perfect isolation is impossible.
Worked example: turn the baseline into a rollback line
Consider an illustrative eight-GPU node test. Before the update, five runs of a 256 MiB all-reduce report a median bus bandwidth of 310 GB/s, with the slowest run at 302 GB/s. After the update, five controlled runs report 291, 294, 292, 293, and 290 GB/s, giving a median of 292 GB/s. The measured change is (292 - 310) / 310 x 100 = -5.8 percent. These numbers are illustrative, not vendor claims, and real results depend on topology, software, thermals, and contention.
Suppose the written rule is: roll back if any correctness error occurs, any run hangs or times out, topology or active-link state changes unexpectedly, or the candidate median is more than 5 percent below baseline at two workload-relevant sizes in two independent test rounds. The example breaches the performance line only if another relevant size also falls beyond 5 percent and the second round reproduces it. That prevents a single noisy observation from triggering rollback while still putting a bound on tolerated loss.
| Signal | Pass condition | Rollback or quarantine condition |
|---|---|---|
| Correctness | Zero reported errors in every required collective | Any mismatch, nonzero error count, hang, or timeout |
| Topology | Expected GPU and NIC paths, active links unchanged | Missing link, changed peer access, or unexplained route change |
| Performance | Candidate median within agreed tolerance at relevant sizes | Reproducible breach in two rounds |
| Stability | Tail and run-to-run spread remain bounded | New outliers or variance beyond the written limit |
Use a fixed diagnostic sequence before changing NCCL knobs
When a candidate fails, first repeat on the same updated node after confirming it is idle. Then compare topology, link status, driver logs, GPU clock and temperature telemetry, CPU binding, NIC selection, and NCCL debug output with the preserved baseline. NVIDIA documents that NCCL relies on detected GPU, PCI, and network topology, while nvidia-smi can report NVLink state. A route or peer-access change is evidence worth explaining before any tuning is attempted.
Next swap one factor at a time: old container on the new driver if supported, new container on the unchanged control, then paired nodes and alternate cables or switch ports under normal operations procedures. Avoid adding persistent NCCL environment overrides merely to recover the headline number. Such a change creates a new configuration that needs its own baseline and may hide the actual regression. The aim is to locate the changed layer, not to make one benchmark green by accumulation of knobs.
Promote a canary only with an exit path
After laboratory acceptance, update a small scheduler-isolated canary pool. Run the same suite there and admit a bounded production workload whose collective mix is known. Compare step time, collective duration distribution, retry or timeout counts, Xid events, link errors, and job completion against an unchanged pool. Define the observation window in work units, such as a fixed number of training steps or completed jobs, rather than an arbitrary quiet hour.
A clean synthetic test cannot prove that every framework, message schedule, or contention pattern is safe. Conversely, a slower application step does not by itself prove a driver regression because input, checkpointing, and host work can move. Keep the prior driver package, node image, and rollback procedure available until the canary completes. The limitation is fundamental: testing samples a workload space. A disciplined baseline and explicit decision rule make that sample auditable rather than exhaustive.