Translate RDMA ready into a testable contract
RDMA support is an end-to-end property. Compatible NICs alone do not establish that the guest image exposes RDMA devices, the intended route is selected, the switches carry the required traffic class, GPU memory can use the path, or all nodes share an adequate fabric. Before renting, write the expected transport, node count, NICs per node, topology, addressing model, isolation, and collective workload into the acceptance criteria.
Ask the provider which layers it owns and which it expects the tenant to configure. Request a topology diagram that identifies leaf groups and oversubscribed uplinks without requiring sensitive facility detail. Also request the supported driver, firmware, CUDA, NCCL, and container combinations. A promise of InfiniBand or RoCE is incomplete if support can later reject the tested software image as unsupported.
Prove device and software consistency on every node
Inventory each allocated host before performance testing. Record GPU and NIC PCI addresses, NUMA nodes, link state and rate, RDMA device names, active ports, GIDs where applicable, MTU, driver, firmware, and library versions. Map each GPU to its nearest NIC. Compare outputs mechanically across hosts because a single replacement node with another firmware or route can destabilize a job that otherwise looks uniform.
Verify that the workload identity can open the required RDMA devices inside its normal container or job sandbox. A privileged host test is not sufficient if production runs unprivileged. Confirm name resolution and rendezvous over the intended management path separately from the data path. Do not disable IOMMU, ACS, firewalls, PFC, or other platform controls to make a test pass; those changes belong to the provider and may weaken isolation or violate its supported configuration.
Test in layers from verbs to GPU collectives
Start with a small host-memory verbs test between one known pair. The linux-rdma perftest suite supplies write, read, send, atomic, bandwidth, and latency programs. Match options on client and server and preserve stderr as well as the summary. Sweep small and large messages, test both directions, and repeat across NIC rails. This verifies a verbs path, not the training application.
Next use a GPU-memory-capable test built and invoked with the provider's supported GPUDirect RDMA path, then run nccl-tests over two nodes and over the full allocation. A host-memory perftest result alone does not prove that registered GPU memory works. Include correctness checks and the collective operations and message sizes the application will use. Finish with a short training step test. Each layer narrows a failure: verbs failure points below NCCL, while passing verbs and failing collectives shifts attention to GPU mapping, libraries, topology, or concurrency.
Sample every rack boundary and then load all nodes
Pair selection should cover same-host paths where relevant, same-leaf pairs, different-leaf pairs, every NIC rail, and the longest documented path in the allocation. Do not choose only adjacent node numbers. Ask the provider to map anonymized hosts to failure or leaf groups so the sample is intentional. For an unknown topology, test a rotating pair schedule and compare distributions to discover clusters of slower paths.
After pair checks pass, generate concurrent traffic from all nodes. Single-pair bandwidth can hide shared uplink oversubscription, hashing imbalance, or congestion control behavior. Run a representative all-reduce at the complete world size and retain per-rank logs and switch-port evidence supplied by the provider. A full-cluster failure with clean pair tests is meaningful evidence, not a reason to average the pairs and declare success.
Use thresholds tied to the workload
Set thresholds before the allocation begins. One example policy is: every required RDMA port is active at its contracted rate; no node differs from the approved version matrix; every topology class completes host-memory bandwidth and latency tests without errors; GPU-memory and NCCL correctness tests pass; and full-scale application throughput reaches at least 90 percent of the agreed reference. Tail latency and run-to-run spread should also have explicit bounds.
Illustrative arithmetic: suppose the reference training run processes 50,000 tokens per second. A 90 percent gate is 50,000 x 0.90 = 45,000 tokens per second. If three measured runs produce 46,000, 44,000, and 45,500, the median is 45,500 but one run misses. Whether that passes depends on the prewritten repeat rule. These numbers are illustrative, not a product claim, and actual performance varies.
| Stage | Evidence | Failure means |
|---|---|---|
| Inventory | Uniform device and version matrix | Allocation is inconsistent |
| Verbs pair | Bandwidth, latency, and error output | Basic RDMA path is suspect |
| GPU memory | Supported direct-path test | GPU-to-NIC path is unproven |
| All-node collective | Correctness and per-rank results | Shared topology or concurrency needs review |
| Application | Useful throughput and stability | Cluster misses the business requirement |
Make provider cooperation part of readiness
Tenants usually cannot see switch counters, queue occupancy, PFC events, ECN marking, cable health, or fabric-manager alarms. Agree on an escalation channel and evidence window before testing. The provider should be able to correlate tenant timestamps and anonymized endpoints with port errors, congestion signals, maintenance, and topology. Without that cooperation, intermittent fabric problems may remain indistinguishable from workload behavior.
For RoCE, ask for the provider's supported traffic-class and congestion-control design, not commands to modify it. NVIDIA's documentation describes PFC and ECN as end-to-end switch and endpoint concerns in relevant designs. Tenant experimentation with these controls on shared infrastructure is unsafe. For InfiniBand, similarly request fabric health and partition confirmation from the operator rather than attempting privileged subnet changes.
Accept only the configuration that was tested
The final report should bind results to allocation IDs, host list, topology groups, images, driver and firmware versions, commands, timestamps, raw outputs, and acceptance decisions. Note any provider remediation and rerun the affected stage plus the stages above it. When using ClusterBid to match capacity, state these measured RDMA requirements alongside accelerator count so offers can be screened for the tested end-to-end behavior.
A passing acceptance test is not a permanent guarantee. Background traffic, node replacement, firmware updates, and application changes can alter results. Keep a short canary that runs at allocation start and after maintenance, with a stricter stop condition for correctness errors than for modest performance drift. The honest limit is that tenant tests observe delivered behavior during a window; they cannot certify every internal fabric condition.
| Decision | Rule |
|---|---|
| Accept | All mandatory stages pass on the contracted topology |
| Conditional accept | Only documented noncritical thresholds miss, with remedy and retest date |
| Reject | Correctness fails, paths are missing, or provider cannot support diagnosis |