Preserve the first failure before retries erase it
An NCCL timeout is a symptom, not a diagnosis. One rank may have stopped entering collectives because its data loader failed, another may be waiting on a peer, or the transport may have stalled. Capture the first error from every rank before restarting. Later watchdog messages often describe consequences rather than the initiating event. Record the job ID, communicator size, rank-to-host mapping, local GPU index, process ID, wall-clock time with timezone, and the first abnormal line for each rank.
Use a single incident clock. Save scheduler events, application logs, kernel messages, and fabric telemetry for a window beginning at least five minutes before the earliest failure and ending after termination. Preserve originals as read-only artifacts, then work from copies. If clocks differ, record the offset instead of silently aligning lines. A defensible timeline is more useful than a large archive whose events cannot be ordered.
Give every rank its own log and identity
For a controlled reproduction, use NCCL logging at the lowest level that answers the question. NCCL_DEBUG=WARN records explicit warnings; INFO is more useful when transport selection or initialization matters. NCCL_DEBUG_SUBSYS can narrow output to areas such as INIT, NET, GRAPH, or COLL. Set NCCL_DEBUG_FILE to a pattern containing host and process placeholders so concurrent ranks do not overwrite one file. Keep the application stack trace and launcher output beside those files.
Do not enable maximum tracing across a full production run by reflex. Extra volume can perturb timing, fill local disks, and make the first signal harder to find. Reproduce with the smallest stable rank count and message pattern that still fails, then expand logging for that case. State every diagnostic environment variable in the bundle, because debug settings and interface overrides are part of the experiment, not incidental metadata.
Reduce the job without changing its meaning
Write a reproduction recipe that another operator can execute without guessing: container digest or image identifier, launch command with secrets replaced by named placeholders, node count, ranks per node, GPU mapping, collective or training step, tensor size, iteration range, and whether failure is deterministic. Include one known-good comparison using the same software where possible. A timeout that appears only after hours needs the last successful step and an estimate of recurrence, not the vague label intermittent.
Use a diagnostic sequence: first repeat the original allocation once; next test the suspected nodes with a representative nccl-tests collective; then remove half the nodes while keeping message size and rank placement explicit; finally swap the excluded and retained halves. Stop when a specific host, link group, scale boundary, or application phase follows the failure. This binary isolation is evidence, but it does not prove root cause if allocation placement or background load also changed.
Capture what NCCL saw, not just inventory labels
Save GPU, NIC, PCIe, NUMA, and NVLink relationships for every distinct node type. NCCL_TOPO_DUMP_FILE can write the topology NCCL detected, while tools such as nvidia-smi topo -m and lspci provide independent views. Also record interface names, consistently redacted address aliases that preserve subnet and endpoint relationships, link state, MTU, and the mapping between ranks, GPUs, and NICs. A provider statement that nodes have the same accelerator model does not establish identical paths.
Version evidence should include the application commit or package lock, framework, CUDA runtime, NCCL library, GPU driver, container digest, OFED or inbox RDMA stack, NIC firmware, and scheduler or orchestration image. Collect versions on every node, not only rank zero. A mixed library mount or replacement host can make a nominally uniform cluster heterogeneous. Prefer command output over a manually typed summary, and record the command used.
Use an evidence matrix to expose gaps
Before escalation, make each artifact answer a question. The matrix below prevents a familiar failure mode in which gigabytes of rank-zero output arrive without the rank that first stopped progressing. It also separates mandatory evidence from useful additions, so collection remains feasible during an incident.
A complete bundle should let the receiver identify the first failing rank, recreate the software and placement, compare good and bad runs, and test whether the failure follows a node or a scale boundary. If any of those questions cannot be answered, label the gap and its reason rather than substituting an assumption.
| Question | Minimum artifact | Pass condition |
|---|---|---|
| Where did progress stop? | Per-rank logs and first-failure timestamp | All ranks covered and clocks related |
| Can it be repeated? | Exact sanitized launch recipe | Two attempts with outcomes recorded |
| What path was selected? | NCCL topology and interface mapping | Every rank maps to a GPU and NIC |
| What changed? | Good-versus-bad version diff | Differences explicit, including none |
| Is hardware implicated? | Kernel, GPU, NIC, and switch event window | Same incident window preserved |
Remove secrets while retaining diagnostic shape
Logs may contain access tokens, object-store URLs with signatures, registry credentials, internal DNS names, user paths, dataset names, and command-line arguments. Do not upload the raw directory and hope the recipient ignores them. Create an allowlisted manifest of artifact types, scan copies for credential patterns, and replace sensitive values consistently, such as TOKEN_1 or HOST_A. Consistent aliases preserve correlations that blanket deletion would destroy.
Keep unredacted evidence only in the approved incident store with restricted access and retention. Hash the originals and sanitized copies so reviewers know which bundle they examined. Avoid collecting environment dumps wholesale; select NCCL, CUDA, scheduler, and network variables that matter. Configuration files should be reduced to relevant keys. Secret removal is part of evidence quality because an unsafe bundle cannot be shared with the people needed to resolve the incident.
Escalate when the bundle supports a falsifiable next step
Use a measurable local rule. For example, escalate when one controlled reproduction has complete rank logs and topology, one comparison run exists, versions are captured on all nodes, and the earliest observed failure is bounded by timestamps from the affected ranks. A five-second window can be a local target when clocks are sufficiently synchronized, but it is not an NCCL requirement. Escalate sooner if repeated attempts risk data or availability, and explicitly mark missing fields. Do not spend hours tuning NCCL variables without recording each change; that creates an unreviewable state and may hide a provider-side defect.
Illustrative example: a 32-rank job times out at 02:14:37 UTC. Rank 19 logs a GPU error at 02:14:34, ranks 0 through 18 report timeout at 02:14:37, and ranks 20 through 31 report it at 02:14:38. A two-node test containing rank 19's host fails twice, while the same test on two other hosts passes twice. That evidence prioritizes the host or its path, but does not prove which component failed. Actual incident timing varies. When comparing rental options on ClusterBid, attach this measured acceptance case to the capacity requirement so replacement capacity is tested against the same workload.
| Decision | Measurable rule |
|---|---|
| Escalate now | Complete evidence plus reproducible failure or credible risk from another run |
| Collect once more | One safe run can fill a named, decision-changing gap |
| Stop reproducing | Risk of corruption, thermal damage, or wider service impact |