Treat rollout capacity as a timeline, not a replica count
A deployment that normally serves on eight GPUs may need nine or ten during an update. The extra demand is not duplication by accident. A rolling controller creates new replicas before it removes old ones when maxSurge is positive, while old replicas may remain alive to finish requests. New processes can also reserve GPU memory before they are ready to receive traffic. For a period, both generations consume accelerators even though only one generation carries most of the load.
Draw a capacity timeline with four states for every replica: scheduled, loading, ready, and draining. Count a GPU as occupied in all four states. Count it as serving only while ready, and possibly while draining if existing streams continue. This distinction prevents the common mistake of comparing desired ready replicas with cluster GPU inventory. The controlling quantity is peak occupied replicas, not the steady-state target shown on a dashboard.
Translate surge settings into physical GPUs
Kubernetes defines maxSurge as how many Pods the controller may create above the desired replica count during a rolling update, rounding percentage values up. MaxUnavailable limits how many desired replicas may be unavailable, rounding percentages down. These are controller rules, not a hard ceiling on live Pods or GPU reservations. Terminating Pods are not counted as available and may continue consuming resources, so total resource use can exceed replicas plus maxSurge until they exit. If each Pod requests one whole GPU, translate every serving, loading, and terminating Pod into one occupied GPU. For multi-GPU Pods, MIG slices, or topology constraints, use the exact schedulable resource shape.
Use a preflight budget that includes steady replicas, maximum surge, the measured maximum number of concurrently terminating replicas, and an explicitly chosen recovery margin. Do not start automatically when compatible schedulable capacity cannot cover that budget, unless the rollout is intentionally configured to wait for draining Pods to release devices. Lowering maxSurge can reduce loading overlap, but it lengthens the update and does not by itself bound resources held during termination. Raising maxUnavailable trades serving capacity for faster replacement and must be validated against load.
| Control | What it protects | Capacity consequence |
|---|---|---|
| Higher maxSurge | Serving capacity while replacements warm | More GPUs occupied temporarily |
| Lower maxUnavailable | Minimum ready capacity | Old replicas remain longer |
| Longer drain grace | Completion of active streams | Old GPUs release later |
| Strict readiness | Traffic from cold or incomplete replicas | Surge replicas wait longer before turnover |
Separate process health from model readiness
A process can answer a shallow health endpoint while weights are still downloading, deserializing, moving to device memory, compiling kernels, or building caches. A readiness probe should represent the ability to accept production inference, not merely the existence of an HTTP listener. Keep liveness conservative during long initialization so the kubelet does not repeatedly kill a healthy loader. A startup probe can protect slow initialization before liveness checks begin.
Readiness should become true only after the model identifier, expected parallel topology, and required workers are available. If the service performs a warmup request, decide whether it is essential to correctness or only improves latency. Making an expensive optional warmup a readiness requirement extends double allocation. Omitting all warmup can send the first user request through compilation. Measure both load-to-ready time and first-request behavior, then state the chosen contract explicitly.
Drain traffic before expecting device release
Pod termination is a sequence, not an immediate GPU return. Kubernetes marks terminating endpoints not ready for ordinary traffic, runs a preStop hook when configured, sends a termination signal, and waits up to the grace period before forcible shutdown. Gateways and clients may still hold streaming connections. The serving process must stop admission, allow bounded work to finish, and then exit. Only after the container exits can its GPU allocation become available to another Pod.
Coordinate three timers: gateway connection draining, application drain deadline, and terminationGracePeriodSeconds. The outer Kubernetes grace period must exceed the application deadline plus shutdown overhead. The gateway should stop new routing before the application deadline starts. If a maximum generation can last 120 seconds but the effective grace is 30 seconds, a nominally graceful rollout will still cut streams or force retries, adding load to the surviving generation.
Worked example: an eight-GPU service on ten GPUs
Consider an illustrative service with eight replicas, one GPU each, maxSurge 25 percent, and maxUnavailable zero. Kubernetes rounds the surge percentage up: 8 multiplied by 0.25 equals 2, so the first step can occupy 10 GPUs. Two new Pods load for four minutes while all eight old Pods serve. When the new Pods become available, two old Pods enter a 90-second drain and may retain their GPUs. The controller can then create the next two replacements because terminating Pods do not count as available. Demand can therefore reach 12 GPUs: six serving old Pods, two terminating old Pods, and four new Pods. On a cluster with only 10 compatible GPUs, the next two replacements remain pending until devices are released.
This scenario has no fault margin and can pause for every drain interval. A hypothetical capacity budget that permits one two-Pod drain cohort plus one recovery GPU is 8 steady + 2 surge + 2 terminating + 1 recovery = 13 compatible GPUs. That is an illustrative planning rule, not a universal requirement; more overlapping terminations require more. If the topology offers only ten, set maxSurge to one or otherwise pace the rollout, verify that serving replicas can absorb any transient imbalance, and expect pending replacements while old Pods drain. ClusterBid can help match the measured temporary shape to suitable capacity.
Observe the transition at request and device layers
Watch desired, updated, ready, available, and terminating replica counts alongside pending Pods and GPU allocations. Add model load duration, readiness transition time, active requests, queued requests, and drain age by revision. Request metrics should include time to first token, inter-token latency, errors, and retry volume. A rollout can appear healthy at the controller layer while long prompts pile up on the old revision or retries double admission pressure.
Use a measurable pause rule for each step. Proceed only when the new replica has been ready for at least two normal traffic windows, its error rate is no worse than the old revision by the agreed tolerance, and fleet queue depth remains below the predeclared ceiling. Roll back if readiness exceeds its historical upper bound or any draining replica reaches 80 percent of its termination grace with active requests remaining.
Plan for scheduler and rollback limits
Nominal free GPU count can overstate usable capacity. A replacement may require a particular GPU model, driver, zone, node affinity, tensor-parallel group, or contiguous MIG profile. Image layers and model weights may also compete for network and local storage bandwidth when several Pods start together. Test scheduling with the real resource requests and placement constraints. Staggered loading can be more reliable than filling every available accelerator at once.
No formula guarantees a safe rollout under hardware failure, a broken readiness implementation, or a new model whose memory footprint differs from the old one. A rollback also needs surge capacity because the previous revision must be recreated after some old Pods have gone. Preserve the old image and model artifacts, cap rollout progress, and rehearse rollback under load. Capacity planning reduces predictable contention; it does not replace a tested abort path.