Cross-Zone GPU Training: Measure the Penalty Before Committing

Cross-zone capacity can look attractive when one zone cannot supply the requested GPUs, but a regional network is not automatically a training fabric. Extra path length and shared infrastructure can affect collective latency, bandwidth, jitter, and failure behavior. The relevant question is not whether two virtual machines can exchange traffic. It is whether the complete job meets a declared throughput and reliability target when its communicating ranks span zones.

Define two candidates with identical accelerator count, software, batch construction, and storage location: a single-zone baseline and the proposed cross-zone placement. Record the exact rank map because tensor parallel, pipeline parallel, and data parallel traffic have different sensitivity. Keep the test inside one region if that is the proposal. Cross-region results answer a different architecture and cost question.

Begin with round-trip latency distributions and sustained throughput between every participating zone pair, in both directions and under concurrency. Report median, p95, p99, minimum, and maximum rather than one average. Test small messages for latency sensitivity and large transfers for steady bandwidth. A compact single-zone placement is a useful control because official cloud placement guidance explicitly connects physical proximity with lower inter-instance latency.

Run long enough to see variation, but do not claim that a short test predicts every production condition. Packet tools establish the network envelope; they do not reproduce GPU synchronization, framework scheduling, or congestion created by all ranks. Preserve timestamps and interface counters so an application slowdown can be aligned with retransmissions, drops, or a zone-pair latency excursion.

Use nccl-tests with the same total rank count and a message-size sweep that covers the application's observed collectives. Run all-reduce, all-gather, and reduce-scatter if the training strategy uses them. Compare a placement where all communication remains within one zone with one where the intended rank groups span zones. NCCL's reported algorithm bandwidth and bus bandwidth are useful, but the final acceptance metric remains application throughput.

Topology matters more than a single pair. If eight workers in zone A communicate with eight in zone B, launch all ranks concurrently. A lone stream may obtain a large share of an uplink that many ranks must divide. Repeat with storage reads and checkpoint writes disabled, then enabled, to identify whether the penalty comes from collectives, shared network demand, or both.

Run the same warm-up, measured steps, global batch, sequence-length distribution, precision, checkpoint policy, and data snapshot. Report samples or tokens per second, median and p95 step time, collective time share, GPU busy time, and failed or retried steps. Do not compensate for a weak network by changing gradient accumulation in only one candidate, because that changes optimization behavior and the amount of synchronization per sample.

Use at least three repetitions per placement after warm-up, and retain per-step distributions. A practical decision rule is to accept cross-zone placement only if median useful throughput is at least 95 percent of the single-zone baseline, p95 step time is no more than 10 percent worse, and no run introduces unexplained collective failures. Those thresholds are policy examples, not universal constants; choose stricter values for costly long jobs.

Cloud providers may meter traffic that crosses availability-zone boundaries. Confirm the applicable service path and both traffic directions using the provider's current billing documentation for the proposed contract. Estimate bytes from observed interface counters during a representative run rather than multiplying theoretical link rate by runtime. Include checkpoint replication, dataset reads, rendezvous traffic, monitoring, and retries, then separate metered bytes from traffic that the provider documents as uncharged.

Illustrative planning example: a billed-direction counter records 6 TB across a zone boundary during a 10-hour run. Using decimal units, 6 TB is 6,000 GB. If the hypothetical charge is $0.01 per GB for that direction, 6,000 GB x $0.01 equals $60 per run, and 20 runs would equal $1,200. This is not a provider quote. Billing direction, binary versus decimal units, free allowances, discounts, and actual prices vary, so replace every assumption with the contract and measured counters.

Providers such as AWS document availability zones as physically separate locations or facilities within a region, but the exact failure model is provider-specific. One synchronous job spanning zones does not automatically become highly available. If every rank is required for every step, loss of one zone can terminate the whole job. Zone diversity helps only when the training system has capacity, checkpoints, rendezvous behavior, and automation that convert separation into faster or safer recovery.

During a controlled nonproduction test, terminate one worker through the scheduler or provider API and measure detection, teardown, replacement placement, checkpoint restore, and return to useful throughput. Do not simulate failure by changing switch, kernel, or RDMA settings on rented hosts. Record whether replacement capacity is available in the required zone and whether the job can resume with a different world size. The provider's infrastructure boundary is not the application's recovery guarantee.

A useful sequence is: validate the single-zone baseline, characterize every cross-zone pair, run full-rank collectives, run the application with normal storage traffic, and finally exercise one worker failure. Freeze the images and configuration between stages. If cross-zone application throughput misses the target, do not infer the exact cause from latency alone; use the layer-by-layer results to decide whether rank placement, communication volume, or storage locality warrants another experiment.

Limit the pilot to enough compute to resolve the decision. For example, require three 60-minute steady-state windows in each placement and one recovery drill. Reject if any acceptance threshold fails twice. The limitation is workload drift: a later model with larger tensor-parallel groups or smaller collectives may respond differently. Keep the benchmark recipe with the capacity request, and rerun it when topology or training strategy changes.

Google Cloud Placement Policies Overview

AWS Regions and Availability Zones

NVIDIA nccl-tests Performance Documentation

AWS VPC Sharing Billing and Metering

FP4 Inference on Blackwell Cost Guide 2026: 8x Claims vs Production Reality

A/B Testing for AI Models: Production Inference Experimentation Framework