Link up is only the first gate
A replaced cable or transceiver can restore carrier, pass neighbor discovery, and carry management traffic while still behaving poorly under sustained GPU communication. Logical success proves that endpoints can establish a link and exchange some data. It does not establish stable throughput, acceptable corrected-error behavior, or clean operation at the intended concurrency.
Treat a physical-layer change as a controlled cluster change. Record the exact ports and components touched, preserve the previous measurements if available, and retest the path from simple link checks through the collective used by the workload. The goal is not to prove that a new component is flawless. It is to show that the changed path returns to an accepted operating envelope without moving the problem elsewhere.
Capture the before state while it still exists
Before replacement, capture link state, negotiated mode, module identity where exposed, switch and host port counters, FEC configuration, corrected and uncorrected error counters, discards, link flaps, and a timestamped traffic test. If the link is too unstable for throughput, record that limitation rather than substituting a remembered number. Take readings at both ends because a receive-side symptom may not appear in the transmitting endpoint's summary.
Also capture the workload-facing baseline: a fixed-size RDMA bandwidth run and a representative NCCL collective across the affected path. Preserve command lines, duration, message size, queue-pair or rank count, node mapping, and software versions. An old result is comparable only if the path and test conditions can be reconstructed.
Make the physical change auditable
Record the removed and installed component identifiers, endpoint ports, change time, and who or what automation performed the work. Confirm that the intended port was changed and that no adjacent link was disturbed. Follow the equipment vendor's handling and compatibility guidance; this article does not replace platform-specific optical, electrical, or safety procedures.
After installation, verify that both endpoints agree on link state and expected negotiated settings. A mismatch in speed, FEC mode, lane state, or breakout mapping is a reason to stop before load testing. Clear counters only if your operating procedure permits it, and save the pre-clear readings first. Otherwise take an explicit starting snapshot and analyze deltas. Photographing labels can help the record, but the authoritative identity should come from the device inventory when it is available.
Read counter deltas, not impressive lifetime totals
Counters need a time and traffic denominator. During a defined test interval, collect starting and ending values for corrected codewords or symbols, uncorrectable errors, discards, recoveries, and link-down events that the hardware exposes. NVIDIA's layer-one troubleshooting guidance specifically uses error counters and FEC-corrected bit evidence when assessing marginal modules, cables, or fiber. Counter names and semantics vary, so use the switch and NIC manuals for the deployed versions.
A rising corrected-error count is evidence that FEC is doing work, not automatically evidence of application loss. Its significance depends on rate, vendor thresholds, link mode, and trend. Any increase in uncorrectable errors, link resets, or packet discards during a controlled run deserves investigation. Absence of exposed errors also does not prove the path meets application throughput.
| Observation | What it establishes | What it does not establish |
|---|---|---|
| Link is up | Basic negotiation and carrier | Sustained error-free throughput |
| Corrected errors rise | FEC corrected observed symbols or codewords | Application loss by itself |
| RDMA bandwidth is stable | A tested network data path is stable | GPU collective mapping is optimal |
| Collective passes | That rank group completed the tested operation | Long-duration stability under all loads |
Run a layered burn-in instead of one heroic test
Start with a short point-to-point test on the changed link, then extend the duration, then add bidirectional or representative concurrent traffic, and finally run the relevant GPU collective. The linux-rdma perftest tools provide duration mode, queue-pair controls, counter reporting, and JSON output. NCCL tests can exercise collectives over chosen message-size ranges. Use the same parameters before and after the change. Let the replacement reach its normal operating temperature before declaring the final interval complete, since a cold start samples only part of the operating state.
An illustrative burn-in might use 5 minutes of single-flow traffic, 20 minutes at representative concurrency, and 30 minutes of repeated all-reduce while counters are sampled every minute. Total active traffic time is 5 + 20 + 30 = 55 minutes. These durations are illustrative, not reliability standards. Actual burn-in should reflect failure history, change risk, thermal settling, maintenance windows, and vendor guidance.
Set the pass rule before viewing the replacement result
Define a pass as a conjunction, not a single green light. An illustrative rule requires no new uncorrectable errors or link-down events, no unexplained discard increase, median large-message throughput within 5 percent of the clean pre-change baseline, and p95 collective time within 5 percent. Thresholds must come from your accepted fleet behavior and risk tolerance, not from this example.
If an illustrative clean baseline for the same test path is 190 Gb/s and the replacement records 184 Gb/s, the decrease is (190 - 184) / 190 x 100 = 3.2 percent, inside the illustrative 5 percent band. If the same run adds an uncorrectable error, it still fails the conjunction. These hypothetical rates are not a vendor claim. Actual throughput and counter behavior vary by link technology, topology, firmware, workload, and measurement duration.
Handle ambiguous results without swapping parts blindly
If errors rise at one receiver, reverse the traffic direction and compare both endpoints. If the symptom follows the component when moved under an approved procedure, the component becomes a stronger suspect. If it stays with the port, inspect the endpoint, cage, lane configuration, and upstream path. These outcomes narrow possibilities but are not infallible component diagnoses.
Shared traffic can make throughput fail while physical counters remain clean. Conversely, a short idle check can look healthy before temperature or load exposes a marginal path. Compare the changed path with an untouched peer of the same design when one exists. That control can expose a rack-wide traffic event or software change that affects both paths, although component age and routing may keep the peers from being perfectly equivalent. Escalate with the raw logs, counter deltas, timestamps, topology, and component records. When selecting replacement capacity through ClusterBid, express the requirement as a repeatable acceptance test rather than only a nominal link type.