GPU RACK POWER DISTRIBUTION ARCHITECTURE
A standard 42U rack of 8x NVIDIA H100 SXM GPUs draws approximately 5.6kW at the GPU level, with total rack power draw reaching 8-12kW including networking, storage, and cooling overhead. B200 GPU racks push this to 12-16kW per rack, and B300 racks with liquid cooling reach 20-40kW per rack. Power distribution at these densities requires 415V three-phase power instead of standard 208V, reducing amperage and copper costs by approximately 30%.
The power chain from utility to GPU includes: utility transformer (13.2kV to 480V), switchgear, UPS (typically N+1 with 5-10 minutes runtime), power distribution units (PDU), rack PDU or busway, and finally the server power supply units. Each stage introduces 1-3% efficiency loss, with total end-to-end efficiency of approximately 90-92% for modern GPU data centers. A 100-rack GPU deployment at 15kW/rack requires approximately 1.7MW of critical IT power and 2.1MW of total facility power at 1.2 PUE.
| GPU System | Rack Power | Voltage | PDU Type | Phase | Racks per 1MW |
|---|---|---|---|---|---|
| 8x H100 SXM | 10 kW | 208V | 3-phase switched | 208V 3-phase | ~83 racks |
| 8x H200 SXM | 12 kW | 415V | 3-phase high-line | 415V 3-phase | ~70 racks |
| 8x B200 SXM | 18 kW | 415V | 3-phase + busway | 415V 3-phase | ~47 racks |
| DGX B300 (liquid) | 40 kW | 415V | Busway + CDU | 415V 3-phase | ~21 racks |
COOLING INTEGRATION FOR HIGH-DENSITY GPU RACKS
Air cooling remains viable up to approximately 20kW per rack with rear-door heat exchangers (RDHx) or cold-aisle containment. Above 20kW per rack, direct-to-chip liquid cooling becomes necessary. GPU liquid cooling uses coolant distribution units (CDUs) that circulate dielectric fluid or water-glycol mixture through cold plates mounted directly on GPU modules. A typical CDU supports 8-16 racks and provides 200-500kW of cooling capacity with supply water temperatures of 45-55 degrees Celsius, enabling year-round economization.
The transition from air to liquid cooling carries a 3-6 month lead time for CDU procurement and facility plumbing modifications. Retrofitting an existing air-cooled data center for liquid cooling typically costs $2,000-4,000 per rack for plumbing, CDU installation, and leak detection systems. Greenfield liquid-cooled facilities cost $1,500-2,500 per rack for cooling infrastructure. total cost of cooling over a 5-year period: air cooling represents approximately 35% of facility OPEX, while liquid cooling reduces this to 20-25% through higher efficiency and reduced fan power.
CABLING AND INTERCONNECT BEST PRACTICES
A fully-populated 8x GPU rack requires: 8-16 power cables (C19 or C21 depending on PSU rating), 8 InfiniBand cables (4x OSFP for NVLink + 4x QSFP for InfiniBand fabric), 2 management network cables (1GbE BMC), and optional storage networking cables (25/100GbE). Cable management at 40kW+ densities is challenging: a B300 rack with liquid cooling requires separate routing for coolant hoses (typically 3/8-inch or 1/2-inch diameter), power cables, and data cables to prevent electromagnetic interference and facilitate maintenance access.
Best practice cable routing uses overhead cable trays for power and fiber, with coolant hoses routed beneath the raised floor. InfiniBand cables should maintain minimum 12-inch separation from power cables to prevent signal degradation at data rates above 400Gbps. Cable labeling should follow a standardized convention: rack row, rack position, device type (GPU/IB/ETH/BMC), and port number. Proper cable management reduces GPU cluster deployment time by 30-40% and simplifies troubleshooting during network issues.
RACK DEPLOYMENT WORKFLOW AND TIMELINE
A typical 100-rack GPU cluster deployment follows this timeline: facility readiness verification (2 weeks), rack and PDU installation (1 week), power distribution commissioning (2 weeks including load bank testing), cooling system installation and pressure testing (2 weeks for air, 3-4 weeks for liquid), network cabling (2 weeks for 100 racks), server installation (1 week for racking), and finally system integration testing (2-3 weeks). Total deployment timeline is 12-16 weeks for turnkey GPU cluster deployment.
The sequential nature of GPU cluster deployment creates bottlenecks that project planning must account for. Power commissioning requires utility coordination with 8-12 week lead times for transformer and switchgear installation. Cooling system pressure testing requires the building management system to be operational. Network cabling must complete before server installation to avoid rework. An experienced deployment team of 4-6 technicians can complete approximately 5 racks per day for air-cooled clusters and 3 racks per day for liquid-cooled clusters.