GPU Cluster Networking Setup: InfiniBand Partitioning and Fabric Management for AI Workloads

Leaf-spine architecture for up to 256 GPUs: 8 leaf switches (each connecting 8 GPU nodes) interconnected by 2-4 spine switches. Oversubscription ratio must be 1:1 for training clusters to avoid NCCL all-reduce bottlenecks. For 512+ GPUs, three-tier topology with super-spines is required.

Routing algorithm: adaptive routing (AR) distributes traffic across paths based on congestion; deterministic routing (MINIHOP) provides consistent latency. For predictable all-reduce patterns, MINIHOP with 16 LMC per leaf provides best balance of performance and utilization.

Structured cabling: each GPU node has 4-8 HCA ports to different leaf switches for redundancy. Color-coded cables (orange=leaf-spine uplinks, blue=GPU node downlinks, yellow=inter-spine). Labels with source-destination port IDs at both ends.

OpenSM runs on two dedicated management nodes with /etc/opensm/opensm.conf: routing_engine=ftree, sweep_interval=5s. HA: master at sm_priority=0, standby at priority=8. Standby monitors via ibsmvote, promotes self within 15-30s if master fails. Set NCCL_SOCKET_TIMEOUT_MS=60000 to survive failover.

NVIDIA UFM (commercial): web UI, REST API, adaptive routing based on real-time congestion feedback, automated firmware upgrades. Runs on dedicated appliance managing up to 3000+ ports. For clusters >512 GPUs, UFM adaptive routing reduces NCCL latency 10-15% under congestion.

Key OpenSM parameters: routing_engine (ftree for fat-tree, updn for custom), log_file with level 0x02 for errors only, sweep_interval (5s for rapid discovery, 30s for stable fabrics). Active-standby configured via --guid flag pointing to master SM port GUID.

PKeys provide network-level isolation: 16-bit value with bit 15 indicating full (0x8000|PKey) or limited membership. Assign one partition per tenant: 0x0001 (production), 0x0002 (research), 0x0003 (inference), 0x0004 (storage). Each GPU node can be member of up to 16 partitions.

OpenSM: /etc/opensm/partitions.conf with format partition { name=production pkey=0x7FFF ipoib=ib0.8001 defmember=<GUID> }. UFM: POST /partitions with name, pkey, member GUIDs. ipoib per partition: 0x0001 gets 10.100.1.0/24 on ib0.8001.

PKey enforcement at switch level prevents cross-partition communication. SM computes separate forwarding tables per PKey. NCCL jobs use NCCL_IB_PKEY=0x7FFF to specify partition for communication.

Adaptive Routing + Congestion Control: AR selects least-congested paths dynamically, CC uses ECN marking to throttle senders when buffers exceed thresholds. Enable on ConnectX-7: mlxconfig set CC_ENABLE=1, DCQCN_ENABLE=1. Parameters: cc_window=16384, cc_threshold=3, cc_timer=3.0us.

Virtual Lane QoS: VL15 for high-priority NCCL, VL8 for inference, VL2 for storage, VL0 for management. SL2VL mapping: SL0->VL15, SL1->VL8, SL2->VL2. NCCL sets NCCL_IB_SL=0 (production) or NCCL_IB_SL=1 (research).

DCQCN parameters: CLR_TARGET=0.92 (92% link utilization before ECN marking), INIT_ALPHA=8 (slower response, less oscillation), RATE_REDUCE_PERIOD=4us. Proper DCQCN tuning reduces tail latency 20-40% during fabric congestion.

Monitoring tools: ibdiagnet --check (daily comprehensive health), iblinkinfo (link state on cabling change), ibqueryerrors (per-port error counters every 5 min to Prometheus), and ibnetdiscover (topology verification). Each feeds into cluster monitoring system.

prometheus-infiniband-exporter: per-port metrics for data rate, xmit_discards, rcv_errors, symbol_errors, link width. Grafana dashboard with fabric utilization heatmap, discard rate for congestion, link state for fault detection.

NCCL debug when performance below baseline: ibdiagnet --check --routing for routing symmetry, ibqueryerrors on switch ports connected to slow nodes, nccl-tests on each node to isolate. If bandwidth <140 GB/s for 8x NDR400, check PKey membership and routing algorithm.