A counter total is not a timeline
A retransmission or error counter proves that a device recorded events according to that counter's definition. It does not show when every event occurred, which job generated the traffic, or whether the application waited. A large lifetime value may predate the slowdown. A small delta may be important if it lands exactly inside a short, latency-sensitive collective. Causality requires ordering, scope, and a plausible mechanism.
Begin by naming the symptom precisely: longer step time, collective timeout, lower throughput, or a tail-latency spike. Record the affected ranks and job phase. Then identify the counter's layer. Physical errors, FEC corrections, switch queue drops, NIC transport retries, and TCP retransmissions describe different recovery mechanisms. Do not combine them into a single generic packet-loss total. Record whether recovery is handled by the link, transport, or application, since each layer exposes a different delay pattern and different counters.
Align clocks and stable identities first
Collect timestamps from the training framework, NCCL logs, host kernel, NIC telemetry, and every relevant switch. Note clock source, timezone, synchronization status, and sampling interval. If clocks differ, create an offset from a shared event such as job launch or an intentional diagnostic marker. A one-second polling interval cannot establish ordering among events separated by milliseconds.
Map ranks to hostnames, GPUs, NICs, switch ports, and rack paths for the incident window. Save job and allocation identifiers because port traffic is shared over time. The useful unit is a tuple such as timestamp, job, phase, rank, host, NIC, switch port, counter name, and counter delta. Without this map, a port spike near a job slowdown is only rack-level coincidence.
Sample deltas instead of comparing screenshots
Read cumulative counters at fixed intervals and subtract adjacent samples. Preserve resets, wrap behavior, and link transitions. NVIDIA's Cumulus guidance specifically recommends observing whether error counters increment over time because errors can be left over from link up and down events. Switch documentation also separates error and drop views, which helps avoid treating congestion drops as physical corruption.
Normalize cautiously. Events per second are useful for short windows, while events per GiB transferred can compare different traffic volumes. Keep the raw delta as well. Zero values may mean no events, an unsupported statistic, the wrong interface, or a sampling gap. Confirm the counter definition in the device and driver documentation used by that fleet.
| Evidence stream | Minimum fields | What it cannot prove alone |
|---|---|---|
| Training trace | Step, phase, duration, rank | Which network event caused delay |
| NCCL or transport log | Timestamp, rank, operation, error | Physical location of a fault |
| NIC counter | Interface, delta, interval, definition | Job ownership on a shared link |
| Switch counter | Port, queue or error type, delta | Application critical-path impact |
| Link log | Port, transition time, negotiated state | That traffic was affected before recovery |
Build a worked incident window
Consider an illustrative job whose median step is 420 ms. Between 14:03:20 and 14:03:40, steps rise to 610 ms. A documented NIC retry counter increases by 480 during that 20-second window, so its rate is 480 / 20, or 24 counter events per second. A switch egress queue records 9,600 new drops, or 480 drops per second, on the mapped uplink. In the next sampled interval, both counters have a delta of zero and step time returns to baseline. These values are hypothetical, not observed measurements, and a counter event need not represent one application message.
This establishes temporal correlation and a plausible shared path, not final causality. Add controls. Did unaffected jobs on the same uplink slow down? Did affected ranks traverse that port? Was offered load high enough to explain queue drops? Did retries begin before step duration rose, or only after traffic backed up? If the counter change starts after the application stall, it may be a consequence. A diagram with 1-second bins often exposes that ordering better than a dashboard average.
Advance through a causality ladder
Level one is coincidence: a cumulative counter is nonzero. Level two is correlation: its delta overlaps the slowdown. Level three is localization: affected ranks share the reporting path while controls do not. Level four is mechanism: logs and counter type explain how delay could arise, such as recovery or queue loss. Level five is intervention: a controlled path change or repaired component removes both the counter delta and slowdown, and restoration reproduces the signature when safe and authorized.
Set a decision rule that requires at least correlation, localization, and mechanism before writing retransmissions as the probable cause. Require intervention evidence before writing confirmed cause, unless reproduction would risk production. This wording preserves uncertainty honestly. A single counter should instead be reported as an observation. The threshold is qualitative because device counters differ, but the evidence requirements are measurable and auditable.
Compare controls across jobs and phases
Training traffic is phase-dependent. Initialization, data loading, forward compute, gradient synchronization, checkpointing, and evaluation produce different network patterns. Aggregate job throughput can hide a 200 ms stall inside one phase. Align counter deltas with collective boundaries and checkpoint windows, then compare the same phase in healthy steps.
Use three controls where possible: the same job before or after the incident, an unaffected job on a different path, and traffic on the same path outside the affected ranks. If retransmission deltas occur during healthy periods at similar rates, the proposed threshold lacks specificity. If slow steps occur without the delta, the counter lacks sensitivity. Either finding can disprove a simplistic rule and improve the next alert.
Close the incident with a reproducible record
Archive raw time series, counter definitions, clock offsets, topology, software and firmware versions, queries, and the final rank-to-port map. State gaps explicitly, including missed samples and counters unavailable at one endpoint. Record alternative explanations such as compute stragglers, storage pauses, congestion without loss, link flaps, or a control-plane event.
The limitation is that observational telemetry may never prove a unique cause in a shared fabric. Sampling can miss short bursts, and counters can aggregate many flows. The defensible outcome may be probable network contribution rather than confirmed root cause. That conclusion is still useful when it directs a controlled test, better instrumentation, or a procurement requirement for per-port telemetry and retention.