Write the failure equation before the topology diagram
A multi-zone diagram proves distribution, not survival. State the number of zones, normal traffic share, usable capacity in each zone, routing behavior, model placement, and what happens to in-flight requests. If three zones each run at 60 percent utilization and one fails, the surviving capacity may absorb the displaced third. If two zones each run at 60 percent and one fails, the survivor faces 120 percent of its nominal capacity before retries.
Use the basic degraded-load ratio: total admitted demand divided by surviving tested service capacity. Set an operating rule before launch. For example, require this ratio to stay at or below 0.80 after loss of any one zone for the critical tier. The 0.80 value is illustrative. It leaves recovery headroom but costs idle capacity. A lower-priority tier may accept shedding or asynchronous completion instead of funding static spare GPU capacity.
Routing and capacity are different promises
Health-based routing can stop sending new work to an impaired zone, but it cannot create GPU workers elsewhere. Scaling after failure also depends on a working control plane, available GPU inventory, model artifact transfer, node initialization, and warmup. AWS describes static stability as provisioning enough capacity in the remaining locations to carry the workload without launching replacement capacity during the failure. That design trades higher steady-state cost for less dependence on recovery-time control-plane actions.
Document whether the service is active-active, active-standby, or a mixture by tier. For active-active, verify topology placement and actual ready capacity, not desired replica counts. Kubernetes topology spread constraints can distribute matching pods across zone labels, but they do not guarantee that each zone has equal GPU types, model readiness, or dependency capacity. The routing assumption and every downstream dependency must survive the same declared failure.
Bound overload before it spreads
An unbounded queue converts a capacity failure into a latency failure and then a recovery failure. Set maximum queue depth or maximum queue age from the request deadline. Reject or degrade work that cannot start in time. Apply admission by workload cost when prompt length, output limit, or model class materially changes GPU time. A request count alone can hide an overload made of unusually expensive requests.
Retries need their own budget. Retry only errors likely to be transient, at one controlled layer, with backoff and jitter, and cap attempts. During overload, retries consume the same scarce capacity as first attempts and can amplify the arrival rate. Protect interactive work from batch queues, and define which traffic sheds first. A partial service that rejects excess work promptly is often more useful than a nominally available endpoint whose queue exceeds every client deadline.
Worked example: one zone disappears
Assume an illustrative three-zone service receives 900 requests per second. Tested usable capacity is 400 requests per second per zone at the latency objective, for 1,200 total. Normal planned utilization is 900 divided by 1,200, or 75 percent. Losing one zone leaves 800 requests per second of tested capacity. If all 900 requests remain admissible, degraded load is 900 divided by 800, or 112.5 percent, so the queue grows by roughly 100 requests each second before retries.
To hold degraded utilization to an illustrative 80 percent, admissible demand is 800 times 0.80, or 640 requests per second. The system must therefore reserve more capacity or shed 900 minus 640, which is 260 requests per second. If each surviving zone instead has 563 tested requests per second, two zones provide 1,126 and 900 divided by 1,126 is about 80 percent. All figures are illustrative; actual throughput must come from the service's workload-shaped load test.
Drill the transition and recovery
A useful exercise removes one zone's inference targets while holding a representative arrival pattern. Observe detection time, routing convergence, admission decisions, retry volume, queue age, p95 and p99 latency, error classification, and dependency health. Confirm the remaining zones already possess the model and tokenizer. Then keep the impairment long enough to expose slow leaks, autoscaler oscillation, quota limits, and exhausted connection pools.
Recovery deserves a separate phase. Do not immediately pour the entire backlog and normal traffic into returning workers. Warm models, pass readiness checks, ramp a bounded share, and drain only requests that remain valuable. Track the time until queue age returns below its objective without breaching live-traffic latency. A drill passes only if it meets declared degraded targets; merely restoring replica count is not a service-level recovery result.
Publish the assumptions operators will need
Put a compact contract beside the runbook: failure unit, expected traffic distribution, tested capacity per ready worker, minimum ready workers per zone, maximum queue age, retry budget, shed order, data consistency assumption, and recovery ramp. Alert on effective surviving capacity and queue age rather than GPU utilization alone. High utilization may be normal, while a rising oldest-message age is direct evidence that service rate trails admitted demand. Page on the predicted time to breach the queue-age objective, and expose the current shed rate so operators can distinguish deliberate protection from unexplained loss. Recalculate surviving capacity whenever a model, batch policy, GPU class, or serving image changes. ClusterBid comparisons can then use the tested degraded-capacity requirement instead of a raw accelerator count.
Zone independence also has limits. Multiple zones can share regional control planes or dependencies, and client routing may be sticky or slow to converge. A multi-zone service is not automatically multi-region, and a multi-region service can still depend on one artifact store or identity system. State those shared dependencies honestly, and include them in exercises where safe simulation is possible. The goal is not to promise that nothing fails; it is to stop one known failure domain from multiplying work faster than the surviving system can finish it.