GPU Data Center Networking: Cross-Connects, Internet Exchange, and Dark Fiber

GPU clusters operate two distinct networks: the compute fabric and the infrastructure fabric. The compute fabric carries NCCL traffic - gradient synchronization, tensor parallelism communication, and data-parallel all-reduce operations - using InfiniBand (IB) or RoCE v2 (RDMA over Converged Ethernet). Each DGX H100 has 8x NDR400 ConnectX-7 network interfaces: typically 4 assigned to the compute fabric and 4 to the storage fabric. The compute fabric demands microsecond latency and zero packet loss, with NCCL ring completion times of 50-200 microseconds for all-reduce on 8 GPUs and 200-500 microseconds across 64 GPUs. Any congestion or packet loss on the compute fabric directly stalls every GPU in the job, wasting thousands of dollars per minute in idle compute.

The infrastructure fabric carries storage traffic (dataset loading, checkpoint writes), inference serving traffic (HTTP/gRPC requests to deployed models), monitoring and management (Prometheus metrics, IPMI, serial console), and external connectivity to cloud on-ramps, internet exchange points, and customer VPNs. Unlike the compute fabric, the infrastructure fabric can tolerate occasional congestion but must support 100-400 Gbps per storage node for checkpoint bandwidth. The two fabrics are physically separate - different NIC ports, different switches, different cabling - to prevent any possibility of storage traffic affecting gradient synchronization. The separation adds 10-20 percent to networking capital cost but prevents a 1-second storage I/O stall from cascading into a 10-minute NCCL timeout across the entire training cluster.

A cross-connect is a physical cable - typically single-mode OS2 fiber - that runs from the colo tenant's cage or suite to another tenant's cage, a carrier meet-me room (MMR), or a cloud on-ramp point of presence. For GPU deployments, the cross-connect density is extreme: a 256-GPU cluster with 16 compute nodes requires 128 compute-fabric cross-connects (NDR400 2x200G), 16-32 storage-fabric cross-connects (100G or 400G each), 4-8 cloud on-ramp cross-connects (100G each), and 2-4 carrier cross-connects for internet transit, totaling 150-170 cross-connects for a single pod. Each cross-connect in a Tier 1 colo costs $200-800 per month, bringing the monthly cross-connect bill to $30,000-135,000 per pod.

Cross-connect types vary by media and speed. Single-mode fiber (OS2) cross-connects are standard for speeds above 10 Gbps because they support longer distances without signal degradation. The most common GPU deployment uses 100GBASE-LR4 or 400GBASE-FR4 optics for infrastructure fabric and passive fiber pairs for InfiniBand compute fabric (the optics are on the NIC and switch ports, not in the cross-connect path). Some colo providers charge a premium for high-density fiber troughs or fiber raceways needed to route 150+ cross-connects per pod. The physical layout must accommodate proper bend-radius (minimum 7.5 mm for OS2 fiber) and cable segregation (separate trays for compute fabric and infrastructure fabric to avoid accidental disconnection during moves, adds, and changes).

Internet Exchange (IX) peering reduces transit costs and improves latency for GPU data centers serving inference traffic. At an IX, the data center connects to a switching fabric shared by hundreds of network operators (ISPs, content providers, cloud platforms) and exchanges traffic with them directly, bypassing for-profit transit providers. Many GPU data centers - particularly those running inference-as-a-service - peer at major IX points: Equinix Internet Exchange (equinix-ix), DE-CIX, AMS-IX, LINX, and regional exchanges. Peering traffic volume for a large GPU inference cluster can reach 100-500 Gbps, which at $0.50-1.50 per Mbps for transit would cost $50,000-750,000 per month - almost entirely eliminated by peering.

The economics favor peering above approximately 10 Gbps of sustained traffic. The IX port cost is typically $1,000-5,000 per month for a 100 Gbps port, plus a one-time membership fee of $3,000-10,000. Remote peering via IX APIs (e.g., Equinix Fabric, Megaport) allows connection from the GPU data center to the IX fabric without physical cross-connects, reducing setup from 10-15 business days to minutes. The peering policy should be open (settlement-free) for content and cloud providers but selective for consumer ISPs, which may not provide symmetric traffic ratios. GPU clusters with significant downstream traffic (model download, dataset distribution) benefit from on-net caching and CDN peering to reduce last-mile bandwidth costs by 30-60 percent.

Dark fiber - leasing unlit fiber strands from a provider and lighting them with the tenant's own optics and DWDM gear - becomes cost-effective for GPU data centers connecting multiple facilities within a metropolitan region. A typical dark fiber IRU (Indefeasible Right of Use) in a major metro costs $200-500 per strand-mile per month on a 15-20 year term with a 2-4 year initial commitment. For a GPU cluster operator connecting three 10 MW data centers across 10 miles of metro fiber, leasing 24 strands of dark fiber costs $48,000-120,000 per month, plus $200,000-500,000 in DWDM equipment per location. The alternative - 100 Gbps lit services from an incumbent carrier - costs $8,000-15,000 per month per 100 Gbps circuit, requiring 4-8 circuits per location for storage fabric connectivity, totaling $32,000-120,000 per month with minimal equipment cost.

The breakeven point for dark fiber versus lit services is typically 4-6 100 Gbps circuits per route or approximately 2-3 years of cumulative carrier costs. Dark fiber provides two additional advantages for GPU clusters: latency certainty (no shared carrier OEO regeneration) and bandwidth scale (a typical dark fiber pair can carry 40-80 channels of 400 Gbps DWDM for a total capacity of 16-32 Tbps per strand pair). For real-time GPU inference workloads that are latency-sensitive below 5 milliseconds between cluster sites, dark fiber eliminates the variable latency introduced by carrier aggregation and optical-electrical-optical conversion. The operational overhead includes managing the optical layer (Raman amplifier tilt, dispersion compensation, OSNR budgeting) and fiber maintenance (locate requests, restoration of accidentally cut fibers).

Network infrastructure represents 8-15 percent of total GPU cluster TCO, a non-trivial fraction that operators often underestimate during initial budget planning. The per-GPU network cost breaks into three categories: compute fabric ($800-1,500 per GPU port for InfiniBand NDR switches and optics), storage fabric ($300-600 per GPU port for 100/400 GbE switches), and external connectivity ($100-300 per GPU per month for cross-connects and transit). For a 1,024-GPU H100 cluster, total networking capital expenditure is $1.2-2.0 million ($1,200-2,000 per GPU) and monthly recurring connectivity costs are $100,000-300,000 ($100-300 per GPU per month).

Cost optimization at the network layer focuses on oversubscription ratios. The compute fabric must remain at 1:1 (non-blocking) for training performance, but the storage fabric can typically tolerate 3:1 to 10:1 oversubscription depending on checkpoint write patterns. External connectivity can run at 20:1 to 100:1 oversubscription for inference and dataset access. Each level of oversubscription reduces switch port count and optics cost by the oversubscription factor. A 1,024-GPU cluster using 4:1 oversubscription on storage saves $150,000-300,000 in switch and optics cost compared to 1:1. The tradeoff is increased latency under load, which must be verified against the application's I/O profile before deployment - checkpoint-heavy workloads like LLM training cannot tolerate storage oversubscription above 3:1 without impacting training throughput.