THE GPU NETWORKING LANDSCAPE
GPU cluster networking has bifurcated into three tiers. NVLink handles GPU-to-GPU within racks at 900 GB/s bidirectional per GPU. InfiniBand NDR400 connects racks at 400 Gb/s per link with sub-1 microsecond latency. Ethernet with RoCEv2 and Spectrum-X reaches 400-800 Gb/s with 3-5 microsecond latency. The choice determines training throughput, scaling efficiency, and per-port cost.
NVLINK: THE INTRA-RACK BACKBONE
NVLink 5.0 provides 1.8 TB/s bidirectional bandwidth per GPU in B200 and B300 configurations, up from 900 GB/s in H100. NVSwitch 5.0 connects up to 576 GPUs in a single NVLink domain with 2.5x all-reduce bandwidth versus H100. This eliminates the networking bottleneck for models fitting within 576-GPU domains. Training throughput improves 1.3-1.6x versus InfiniBand-connected H100 clusters of equivalent scale.
INFINIBAND: THE TRAINING STANDARD
InfiniBand NDR400 delivers 400 Gb/s per link with sub-1 microsecond latency and 0.00001% packet loss. Lambda's CPO integration reduces power by 30% and latency by 15%. Scaling efficiency reaches 90-95% for models up to 1T parameters across 1,024 GPUs. Cost is $1,500-2,500 per port versus $500-800 for high-end Ethernet.
| Feature | NVLink 5 | InfiniBand | Spectrum-X |
|---|---|---|---|
| Bandwidth/port | 1.8 TB/s | 400 Gb/s | 800 Gb/s |
| Latency | <100 ns | <1 us | 3-5 us |
| Max GPUs/domain | 576 | 65,536 | 32,000 |
| Cost/port | N/A | $1,500-2,500 | $600-1,200 |
| Scaling eff. | 95-98% | 90-95% | 80-90% |
ETHERNET: THE DARK HORSE
CoreWeave deployed 102.4 Tb/s Spectrum-X Ethernet with RoCEv2 achieving 95% of InfiniBand training performance at 50-70% cost. Ethernet advantages include existing operational tooling, broader vendor ecosystem, and lower training costs. For clusters under 512 GPUs, modern Ethernet delivers 85-95% of InfiniBand scaling efficiency at 40-60% lower networking cost.
COST COMPARISON
For a 256-GPU cluster: InfiniBand networking adds $400,000-600,000 ($1,500-2,500/port) versus $150,000-250,000 for Spectrum-X Ethernet. The networking cost premium is 8-15% of total cluster cost for InfiniBand versus 3-6% for Ethernet. For clusters under 1,000 GPUs, Ethernet savings of $200,000-500,000 can fund 20-50 additional GPU-hours per day.
SELECTION FRAMEWORK
Choose NVLink-only for single-rack deployments (8-72 GPUs) without inter-rack training. Choose InfiniBand for clusters above 512 GPUs or models above 100B parameters requiring 95%+ scaling efficiency. Choose Ethernet for clusters under 512 GPUs, budget-constrained deployments, or teams with existing Ethernet operational expertise. Most production deployments combine all three: NVLink within racks, InfiniBand or Ethernet between racks.