Define warm as usable capacity
A warm GPU worker is not merely a running virtual machine or a pod with an allocated device. It has the required image and model revision, a usable GPU context, loaded weights, healthy dependencies, and a readiness result that permits new traffic. That definition prevents infrastructure dashboards from counting capacity that still needs a long model download or compilation step. Track warm-ready, warming, busy, draining, and unhealthy states separately.
The pool exists to cover demand arriving during scale-out delay. It is insurance purchased with idle GPU time, memory residency, and operational complexity. It should not conceal an application that cannot initialize reliably. Measure the interval from a scale decision until a worker passes the same readiness check used by routing, then calculate the warm reserve from arrivals during that interval and the service rate of a ready worker.
Model the startup gap, not the daily peak
Use production traces to form rolling arrival counts over the measured cold-start interval. For each interval, subtract capacity already available and convert the positive remainder into workers. A percentile of that distribution provides a starting reserve. Keep traffic classes separate if they use different models or cannot share workers. A single daily maximum often overstates reserve because slower autoscaling can cover a sustained ramp after the initial gap.
Do not assume every allocated GPU provides equal capacity. Estimate successful requests per second for a defined input and output mix, with the same batching and latency objective used in production. Recalculate after model or scheduler changes. The warm-pool model is only as sound as this service-rate estimate; a pooled average can be unsafe if one model revision is much slower than another.
Work an illustrative burst from arrivals to GPUs
Assume, illustratively, that baseline traffic consumes four ready workers and each worker sustains 3 successful requests per second for the target mix. Cold scale-out p95 is 100 seconds. A burst adds 8 requests per second for 45 seconds, creating 8 x 45 = 360 incremental requests before the burst ends. Two otherwise idle warm workers add 2 x 3 x 45 = 270 requests of incremental service during that burst, leaving 90 requests. At 3 requests per worker-second, that backlog is 30 worker-seconds, or 15 elapsed seconds if both warm workers remain available after arrivals normalize.
Three warm workers would add 3 x 3 x 45 = 405 requests of service, enough for that simplified burst. These are illustrative assumptions, not performance claims, and actual throughput and prices vary. The example also ignores queueing within batches and response-length variation. A decision rule could require the warm pool to absorb the p95 incremental work arriving before cold capacity is ready while keeping predicted oldest-request age below the service objective.
Price idle time against avoided delay
Report warm capacity as GPU-minutes reserved, useful work completed, and delay breaches avoided. Suppose an illustrative GPU costs $2.00 per hour and one extra worker remains idle for 18 hours in a day. Its idle planning cost is 18 x $2.00 = $36 per day. If it prevents 1,800 deadline breaches, the simple planning ratio is $36 / 1,800 = $0.02 per avoided breach. All figures are hypothetical; actual prices and business impact vary.
That ratio is not a universal optimization target. Some requests have different value, and a breach may not be prevented solely by the reserve. Compare policies through trace replay or controlled traffic rather than assigning every improvement to warm capacity. Include model-store traffic, host reservation, and minimum billing units where they apply, but do not import a live provider price into an evergreen operating rule.
Let the floor follow predictable risk
A fixed minimum is simple and robust. A scheduled floor is often better when demand has reliable daily structure: raise the minimum before a known arrival window and lower it only after the risk period. Event-driven systems such as KEDA expose a minimum replica count and cooldown behavior, while Kubernetes HPA supports scale behavior and stabilization. Verify the exact semantics and version in use, especially around scaling to zero and which controller owns each range.
Start with a conservative rule that operators can audit. For example, retain two warm-ready workers whenever the trailing 28-day p95 incremental arrival count during cold-start windows requires at least two, and change the floor only after seven consecutive daily evaluations agree. This is an illustrative governance rule. It resists one anomalous day while still adapting. Revisit it after major traffic, model, or startup-time changes.
Drain without accepting work too late
Scale-down must stop new admissions before process termination. Mark the worker draining, remove it from new-request routing, wait for the router's view to converge, and allow bounded in-flight work to finish. Kubernetes describes graceful termination, including preStop hooks and a total termination grace period. The hook consumes part of that period, so the configured grace must cover both routing delay and application shutdown, with margin.
Long streaming generations complicate the boundary. Choose an explicit maximum drain duration and define whether remaining requests may finish, migrate, retry, or fail. Retrying can duplicate expensive work or user-visible tokens, so it is not automatically safe. A measurable drain rule is zero new admissions after the drain timestamp, at least 99 percent of in-flight requests completing within the grace interval, and a labeled count for every forced termination.
Protect the reserve from correlated loss
A replica minimum does not guarantee availability if all warm workers share one node, rack, zone, registry path, or model-store dependency. Apply topology spread constraints for the failure domains represented by reliable node labels, and use disruption controls for voluntary evictions, then separately test involuntary node loss. A PodDisruptionBudget does not protect against every involuntary disruption. Readiness must fail closed when the model or GPU is unusable, and capacity monitoring should count routable ready replicas rather than assuming the configured minimum is available.
Use requested replicas, scheduled replicas, ready replicas, and routable replicas as separate signals. Alert if routable capacity stays below the warm floor longer than one control interval. During deployments, surge capacity should preserve the reserve while old workers drain and new ones warm. Otherwise a routine rollout consumes the insurance precisely when a traffic burst may arrive.
Know when warm capacity cannot help
A small reserve cannot absorb an unbounded flash crowd, a region failure, or requests for a model not resident in the pool. It can also waste money when admission control or a short queue would meet the user objective at lower cost. Validate the policy monthly with cold-start distributions, burst traces, utilization, queue age, and breach counts, and keep a documented maximum beyond which requests are rejected or degraded.
The final operating rule should combine a minimum floor, a maximum fleet size, a queue-age trigger, and a drain procedure. If observed bursts regularly exhaust the reserve before new workers become ready, first reduce cold-start time or raise the floor based on arithmetic. If warm workers remain idle while objectives are met for several review periods, test a one-worker reduction. Warm capacity is a bounded latency hedge, not proof that autoscaling is solved.