Contention is a time series, not a slow-run label
A distributed GPU job can slow down when another workload shares a NIC, switch port, uplink, buffer pool, or routed fabric path. The symptom may be lower average collective throughput, wider step-time tails, or bursts of stalls. None of those observations proves multi-tenant contention by itself. A slow rank, storage pause, retransmission, route change, or host scheduling problem can produce a similar trace.
The useful test aligns workload performance, competing traffic, and fabric telemetry on one clock. Establish a quiet baseline, introduce a controlled competitor where permitted, remove it, and look for a reversible change. In a provider environment where tenants cannot generate arbitrary cross-traffic or inspect switches, request timestamped provider evidence rather than treating missing visibility as proof of isolation.
Define what is dedicated and what is shared
Ask separately about GPU hosts, NICs, top-of-rack ports, leaf-to-spine uplinks, queueing resources, and congestion domains. A dedicated server does not imply a dedicated fabric path. Conversely, a shared switch does not establish harmful contention if capacity, routing, and traffic controls keep the tested workload within its service envelope.
Map each rank to GPU, NIC, switch-facing port if disclosed, and rack or failure domain. Record whether traffic stays within one rack or crosses an aggregation boundary. Ask how equal-cost paths are selected and whether flows can move during a job. The provider may not disclose a full topology for security or operational reasons, but it should be able to state the isolation boundary and supply relevant health evidence.
Build a quiet, competed, and recovery sequence
Run the same fixed collective or short training segment in three phases. Phase A measures the best available quiet window. Phase B adds authorized competing traffic on the suspected shared boundary. Phase C removes the competitor and repeats the original workload. Keep ranks, message sizes, process placement, clocks, software versions, and duration unchanged.
Use more than one concurrency level because contention may appear only after aggregate offered load reaches a shared limit. The linux-rdma perftest package supports duration runs, multiple queue pairs, bidirectional modes, JSON output, and performance counter reporting. NCCL tests supports multi-process and multi-GPU collective runs over controlled message sizes. These tools test different layers, so compare each tool with itself across phases. Repeat the three-phase sequence in a different window to reduce the chance that an unrelated event happens to align with Phase B.
| Phase | Traffic condition | Interpretive value |
|---|---|---|
| A | Target job alone as far as observable | Establishes a time-local baseline |
| B | Target job plus authorized competitor | Tests sensitivity to shared load |
| C | Competitor removed | Checks whether the change is reversible |
Measure variation, not only means
Record median, p95, and p99 collective duration or step time, plus achieved throughput for equal work. Capture per-rank results because one path can pace the entire synchronous job. Sample NIC, RDMA, and switch counters at the start and end of each phase, including discards, congestion notifications, pause behavior, retransmission-related counters, and port utilization where the platform exposes them.
Counter interpretation depends on the fabric and configuration. In a RoCE deployment, ECN and priority flow control can be part of congestion handling if they are enabled, but an ECN mark or pause counter increment is not automatically a root cause. For InfiniBand, port congestion and error counters need topology and traffic context. Ask the provider to explain the exact counter definitions and collection interval. Aggregated daily charts are usually too coarse for a five-minute stall.
Calculate a contention penalty against equal work
Consider an illustrative all-reduce sequence whose median duration is 80 ms in Phase A, 104 ms with a competitor in Phase B, and 82 ms after removal in Phase C. The competed penalty is (104 - 80) / 80 x 100 = 30 percent. Recovery is close to baseline because (82 - 80) / 80 x 100 = 2.5 percent. This reversible pattern supports contention as a diagnostic possibility, especially if relevant port utilization or congestion counters rise in the same interval.
The numbers are illustrative, and actual outcomes vary with topology, routing, collective, message size, and background load. A predefined rule might flag a path when Phase B worsens median duration by more than 10 percent or p95 by more than 20 percent, and Phase C returns within 5 percent of Phase A. Those thresholds are an example operating policy, not a universal standard. Repeat on another time window before generalizing.
Request provider evidence that can falsify a theory
A useful provider response identifies the affected endpoints and time window, states whether the path or congestion domain is shared, provides port utilization and relevant error or congestion counter deltas at sufficient resolution, notes route or maintenance events, and supplies a comparable uncontended result. Ask whether counters reset, wrap, or aggregate multiple tenants. Raw values without units and collection intervals are difficult to evaluate.
Also request the service's performance acceptance method: test tool and version, topology scope, message sizes, concurrency, duration, warmup, repetitions, and percentile or variance rule. A screenshot of one successful ping or link-up state does not address sustained collective traffic. Through ClusterBid, matching capacity to a measured requirement is clearer when the requirement includes this evidence contract rather than only a nominal NIC speed.
Separate contention from path faults and host noise
If performance drops with the competitor and recovers afterward while link-error counters stay clean, capacity contention is more plausible than a physical fault. If errors or recoveries rise regardless of competitor traffic, inspect the link. If only one rank degrades while its NIC remains quiet, inspect CPU affinity, GPU health, data loading, and rank placement. If Phase C stays slow, background conditions may have changed or the test may have warmed a different system component.
The limitation is fundamental: a short controlled test samples only specific routes and time windows. It cannot prove future noisy-neighbor behavior or reveal undisclosed internal topology. Long-running traces, repeated windows, and a contractual or operational performance envelope provide stronger evidence. Keep the exact timestamps and raw outputs so the provider can correlate them with telemetry that tenants cannot see. Where controlled competing traffic is prohibited, compare naturally busy and quiet windows without claiming that their difference is causal.