Choose the signal by the question
Queue age and device utilization answer different questions. The age of the oldest eligible request asks how long admitted work has waited. For the commonly exported NVIDIA GPU utilization metric, utilization is the percentage of a sample period during which one or more kernels executed; it is not the percentage of theoretical compute capacity delivered. Queue age can expose an approaching latency miss before a saturated service completes enough requests to change throughput metrics. Utilization shows device activity, but it is a lagging and workload-dependent signal rather than a direct measure of user pain.
For bursty online inference, use queue age as the primary scale-up signal when requests enter a durable, observable queue and have a clear deadline. Use device utilization as corroboration and as a saturation guardrail. For batch jobs, queue age may be the service objective itself, while utilization helps distinguish capacity shortage from admission, scheduling, or data problems. Neither signal should scale blindly when the service is failing to dispatch work.
Understand what utilization hides
An average device utilization can conceal alternating idle and busy periods, imbalance across replicas, and work that occupies the GPU without producing acceptable requests. Dynamic batching can raise utilization while an oldest request waits behind incompatible shapes. A stuck kernel or retry loop may also look busy. Conversely, short kernels separated by CPU preparation can deliver adequate latency at modest utilization. Always pair the gauge with request rate, successful completion rate, queue depth or age, and per-replica skew.
Aggregation is a policy choice. A fleet mean lets idle replicas cancel hot ones. A maximum can react to a single damaged or imbalanced worker. Prefer a high percentile across ready replicas plus an imbalance alarm, and exclude warming or draining replicas from the capacity denominator. Document the sampling interval used by the exporter and the controller query window, because a 60-second average cannot preserve a five-second spike.
Queue age leads, but can still lie
Oldest-request age increases as soon as work waits, making it a useful leading indicator under a burst. It also reflects head-of-line blocking, priority policy, and poisoned requests. A single request that cannot be scheduled can keep the oldest age high while normal traffic flows. Segment the metric by service class and eligibility, and cap or quarantine repeatedly failing work. Queue depth is a useful companion because ten old requests and ten thousand old requests require different responses.
Measure age at one authoritative queue boundary with a monotonic clock where possible. If producers stamp wall-clock time and consumers run on different hosts, clock error becomes apparent queue delay. Exclude work intentionally deferred by a scheduled start time. A diagnostic rule is: if oldest eligible age exceeds its threshold while arrival rate is above successful departure rate for two consecutive control windows, scale out; if departure rate is near zero, enter a failure path before requesting more GPUs.
Translate delay into a replica request
Consider an illustrative service with eight ready workers. Each sustains 2.5 successful requests per second for the measured request mix, so observed service capacity is 8 x 2.5 = 20 requests per second. A burst arrives at 32 requests per second for 30 seconds. The backlog grows at 32 - 20 = 12 requests per second, reaching 360 requests. If arrivals then return to a normal 20 requests per second and consume all original capacity, draining that backlog within 60 seconds needs 360 / 60 / 2.5 = 2.4, rounded up to three additional ready workers.
Every number is illustrative and actual capacity varies with prompts, outputs, batching, and failures. Add startup delay: if a new worker takes 90 seconds to become ready, it cannot help a 30-second burst unless prediction or a warm pool acts earlier. That is why a queue-age controller needs a forecast or minimum capacity for short bursts. Validate the calculation by replaying the real request-length distribution, then use successful work per ready replica rather than a peak benchmark.
Set warmup and cooldown explicitly
A newly created pod should not count as capacity until readiness passes. During image download, model load, compilation, and cache population, its GPU metric may be missing or misleading. Kubernetes HPA accounts for not-yet-ready pods and missing metrics in its algorithm, but custom controllers must implement an equivalent conservative rule. Record requested, starting, ready, and draining replicas separately, and suppress repeated additions only when already requested capacity is expected to cover the measured deficit.
Scale-down needs a longer view than scale-up. Kubernetes supports stabilization windows and scaling policies, and chooses the largest desired replica count when multiple HPA metrics are configured. An illustrative policy is fast scale-up after two 15-second breaches, followed by a 10-minute period with queue age below 20 percent of its threshold before removing one worker. These are planning values, not defaults. Cooldown must exceed the typical burst gap or the fleet will repeatedly pay startup cost.
Add saturation and failure guardrails
Set a maximum replica count from verified quota, schedulable GPUs, registry capacity, and downstream limits. Alert when desired replicas exceed that ceiling, because the controller is then reporting unmet demand rather than solving it. Also protect dependencies: more inference workers may amplify model-store traffic, CPU tokenization, or response writes. Scale-out should pause or rate-limit when new replicas repeatedly fail readiness, since another wave can deepen contention.
Use a three-way classification. Queue age high plus utilization high suggests genuine capacity pressure. Queue age high plus utilization low suggests dispatch, CPU, dependency, batching, or metric problems. Queue age low plus utilization high may be efficient operation, provided latency and errors remain within objectives. No classification proves root cause, but it directs the first investigation. Require successful request rate and error rate to remain valid before utilization is allowed to justify more capacity.
Run the controller as an experiment
Before enabling automatic writes, replay the intended queries against historical data and log the replica decisions in shadow mode. Include flash bursts, gradual ramps, poisoned requests, exporter gaps, node loss, model rollout, and an exhausted GPU quota. Compare avoided queue-delay minutes with added idle GPU minutes. A controller that reacts correctly to a clean load test may still oscillate when readiness takes longer during a fleet-wide pull.
Promote with measurable rules: no more than one scale direction change per chosen cooldown period, fewer than 1 percent of eligible requests breaching the queue-age objective in the evaluation window, and no scale-up caused solely by a missing metric. Those thresholds are local examples and must be tuned. The limitation is fundamental: reactive autoscaling cannot erase startup time. When burst duration is shorter than warmup, reserved warm capacity or admission control remains necessary.