Two traffic classes can make one pool look undersized
Interactive and batch inference ask the same hardware for different things. A chat request cares about queue delay, time to first token, and steady delivery. A document enrichment job may care only that ten thousand records finish before morning. When both enter one first-come queue, a burst of long batch generations can occupy decode slots while short interactive work waits. The visible symptom is a latency spike, but the operator may misread it as insufficient aggregate GPU capacity.
Start by labeling every request at admission, not after it finishes. Record class, arrival time, queue entry and exit, prompt tokens, generated tokens, completion reason, and cancellation. Keep separate percentile views for queue time, time to first token, and end-to-end latency. Aggregate GPU utilization cannot reveal which class caused a wait. The useful question is whether interactive latency deteriorates while the batch deadline still has slack.
Choose the smallest isolation boundary that holds the objective
Queue isolation and dedicated GPU pools solve different parts of the problem. Separate queues preserve class identity and let an admission controller choose which request runs next, but the classes can still interfere inside a shared server through memory pressure, batching, and long decode occupancy. Dedicated pools create a stronger boundary and a clearer capacity budget, at the cost of idle fragments when one class is quiet.
NVIDIA Triton documents instance groups as the control for how many model executions exist and which GPUs host them. Its rate limiter can postpone model-instance execution and apply priorities across loaded models. Those mechanisms can support a shared-pool design, but they do not create a business policy. The operator must still decide how much interactive capacity is protected, whether batch work may borrow it, and how borrowed work is drained when interactive arrivals return.
| Design | What it isolates | Best fit | Main failure mode |
|---|---|---|---|
| Separate queues, shared workers | Admission order and queue budgets | Bursty classes with similar model and memory shape | A running batch request still blocks scarce execution capacity |
| Dedicated pools | GPU execution and memory capacity | Strict latency objective or highly different request shapes | Stranded capacity during asymmetric demand |
| Reserved pool with controlled borrowing | Baseline capacity plus temporary utilization | Predictable interactive baseline and flexible batch deadline | Poor spillback rules turn borrowing into hidden overcommit |
Write a class-specific contract before tuning schedulers
A workable interactive objective needs a percentile, a window, a load range, and a precise clock. For example: while admitted interactive load is at or below 18 requests per second, at least 99 percent of successful requests must start streaming within 700 milliseconds over a rolling 15-minute window. Add a separate bound for time per output token if streaming smoothness matters. Do not hide rejected or timed-out requests by computing the percentile only over survivors.
Batch work needs a completion objective rather than a borrowed interactive target. An example is: 98 percent of jobs admitted before 20:00 must finish by 06:00, with no single job waiting more than two hours before its first attempt. These values are illustrative, not measured recommendations. The important separation is that interactive traffic consumes a latency budget while batch traffic consumes a deadline budget. A scheduler can then trade only the slack that actually exists.
Worked example: lend capacity without lending the SLO
Consider an illustrative eight-GPU service running one model. Four GPUs are the protected interactive floor, two are the batch floor, and two form a flex band. During a quiet interval, interactive demand uses three GPUs and batch backlog is 9,600 GPU-seconds. The batch deadline is four hours away, so the required average batch service rate is 9,600 / 14,400 = 0.67 GPU-seconds per second. Two batch GPUs provide 2 GPU-seconds per second before efficiency losses, so the backlog has substantial slack and does not justify borrowing protected capacity.
Now suppose backlog rises to 36,000 GPU-seconds with four hours remaining. Its simple required rate is 36,000 / 14,400 = 2.5 GPU-seconds per second. Let batch borrow one flex GPU only if interactive p95 queue time has stayed below 200 milliseconds for ten minutes, interactive admission is below 70 percent of its tested safe rate, and batch deadline slack is under 25 percent. When either interactive threshold is breached, stop admitting new batch work to that GPU immediately and require it to drain within a tested 30-second bound. Cancel running batch work only if the backend actually supports prompt cancellation and the work is safe to retry. The numbers are illustrative; model behavior and batching efficiency vary.
Use an interference test as the purchase gate
Run a three-stage diagnostic with the same request-shape traces. First, replay interactive traffic alone across the expected load range and find the highest admitted rate that meets its latency and quality criteria. Second, run batch alone and measure completed useful work per GPU-hour. Third, combine them under the proposed policy while preserving their arrival patterns. Compare interactive latency, rejection rate, batch completion, queue depth, and generated tokens per GPU-second across all three stages.
Use a measurable decision rule: do not add GPUs for mixed-pool saturation if isolation restores the interactive objective and batch still meets its deadline at the current fleet size. Consider more capacity only when the isolated design misses either objective at the forecast load after a representative soak, or when the protected floor leaves insufficient failure headroom. ClusterBid is useful at that point for matching the measured pool requirement to available capacity, rather than starting from a utilization chart.
Spillover is a reversible exception, not a default mode
Kubernetes priority can influence scheduling order, and preemption can remove lower-priority Pods when a higher-priority Pod cannot schedule. The official documentation also notes that victim termination takes time and that a nominated node is not a guarantee of immediate placement. For latency-sensitive inference, pod-level preemption is therefore a coarse recovery mechanism. Keeping warm interactive workers ready is usually more predictable than expecting an evicted batch workload to free a GPU instantly.
The honest limitation is that static class boundaries cannot remove every source of interference. Shared CPUs, network paths, tokenizers, storage, and model gateways may remain common. Request lengths can also shift after a product change. Review the policy when p99 queue time rises, batch deadline slack shrinks, or spillovers occur more than a chosen weekly budget. If the flex band is continuously assigned to one class, the original partition is no longer describing demand and should be resized from fresh traces.