Count reserved tokens, not request envelopes
A request counter treats a 400-token chat turn and a 60,000-token document review as equivalent, although their prefill work and potential cache occupancy differ sharply. Admission control should therefore operate on at least three dimensions: input tokens, requested output tokens, and requests already waiting. Tokenize before admission with the exact production tokenizer. Reject malformed requests and impossible context lengths before they reach the GPU-facing scheduler.
The useful admission reservation is input tokens plus the permitted output ceiling, not an assumed average output. Hugging Face Text Generation Inference describes max total tokens as a per-request memory budget and max batch prefill tokens as a limit on the compute-heavy prefill operation. These engine limits are not a complete admission controller, but they support a practical split: account conservatively at the gateway, then release the gateway reservation when a request finishes or is canceled.
Give interactive and long-prefill work separate lanes
Create an interactive lane for short, latency-sensitive turns and a bulk lane for long document prompts. The cleanest isolation is separate replicas or GPU pools because a large prefill can otherwise pause or delay decode work. If infrastructure must be shared, use distinct queues, explicit priority, and a cap on the fraction of each scheduling interval that bulk prefill may consume. Priority without a bulk floor can starve long jobs, so reserve a small, measured share for them.
Classify by declared service intent and tokenized size, never by an untrusted client priority alone. For example, interactive might require input at or below 4,096 tokens and requested output at or below 1,024 tokens. Everything larger enters bulk. These are illustrative thresholds, not universal recommendations. Publish the lane assignment in response metadata so callers can understand why two requests to the same model receive different latency treatment.
| Lane | Admission budget | Queue behavior | Client contract |
|---|---|---|---|
| Interactive | Small input and output caps | Short bounded queue, higher priority | Fast failure, brief retry |
| Bulk | Larger token reservation | Separate queue, controlled prefill share | Asynchronous or longer deadline |
| Overflow | Beyond normal bulk limit | No GPU queue entry | Summarize, split, or use dedicated capacity |
Turn capacity into an admission equation
Maintain counters per model replica: reserved total tokens, queued prefill tokens, active requests, and queue age. Admit an interactive request only when its total-token reservation fits the configured safe budget and its predicted queue delay remains below the lane deadline. Admit a bulk request only when both the bulk reservation and the next prefill-window allowance fit. A hard concurrent-request cap remains valuable because even small requests consume CPU, network, tokenizer, and bookkeeping resources.
Worked example, with entirely hypothetical numbers: a replica has a tested gross ceiling of 96,000 reserved tokens before a policy guard band. Active requests reserve 58,000, the interactive queue reserves 8,000, and a guard band equal to 10 percent of the gross ceiling is 9,600. Available reservation is 96,000 minus 58,000 minus 8,000 minus 9,600, which equals 20,400 tokens. A prompt of 14,000 tokens requesting up to 4,000 output tokens reserves 18,000 and fits. A 19,000-token prompt requesting 4,000 reserves 23,000 and does not. Actual capacity varies with model, serving engine, precision, parallelism, and hardware.
Reject before the queue becomes the outage
An unbounded queue converts overload into hidden latency and holds connections, memory, and caller deadlines open. Set a maximum queue size for each lane and a maximum residence time. NVIDIA Triton documents per-priority queue policies with maximum queue size and timeout actions named REJECT and DELAY. DELAY can postpone a timed-out request indefinitely behind eligible requests, so it is not equivalent to rejecting overload. The gateway should also refuse work when its own token reservation or deadline estimate says the request cannot finish usefully.
Use a measurable rule: when interactive queue age at the 95th percentile exceeds its service objective for three consecutive one-minute windows, stop admitting bulk work to shared replicas. If it remains high for two more windows, shed the lowest-priority interactive arrivals until recovery. Restore bulk admission only after the same metric stays below 70 percent of the objective for five windows. The exact windows are illustrative; the important properties are a defined signal, hysteresis, and a documented recovery condition.
Return a stable rejection and retry contract
Use HTTP 413 when the tokenized request exceeds the endpoint's immutable input or total-token limit. Use 429 when the request is valid but the lane has exhausted its queue or reservation budget. Use 503 only when the serving capacity is unavailable rather than merely busy. Return a machine-readable reason such as context_limit, queue_full, token_budget_exhausted, or deadline_infeasible, plus the accepted maximums and a request identifier. Do not silently truncate a prompt at this layer.
For transient 429 responses, include Retry-After based on a conservative queue-drain estimate, rounded upward. Clients should apply randomized exponential backoff and honor an absolute request deadline. A retry must not jump priority or bypass accounting. Where the endpoint implements idempotency keys, clients should reuse one for retries of an operation that may trigger downstream side effects. Stop retrying 413 responses unchanged. For bulk work, an accepted asynchronous job with polling is often clearer than holding a synchronous connection through a long queue.
Diagnose overload in the order it occurs
Start with arrival metrics: requests by lane, input-token histogram, requested-output histogram, and rejection reason. Then inspect queue depth, oldest age, and queued prefill tokens. Finally correlate time to first token, inter-token latency, batch composition, cache use, and GPU utilization. TGI exposes input-length and batch-token metrics as well as separate prefill and decode measurements. Those distinctions help test whether long prefills, decode saturation, or an upstream bottleneck is the likely cause.
Use this diagnostic sequence. First, confirm tokenization and lane classification. Second, check whether rejection rises before queue age, as intended. Third, compare interactive latency with bulk prefill activity. Fourth, replay a sanitized production length distribution at controlled arrival rates. Fifth, adjust one budget at a time. High GPU utilization alone is not proof of healthy service, and low utilization with a growing queue may indicate tokenizer, transfer, memory, or scheduler constraints rather than insufficient GPU compute.
Admission control cannot manufacture capacity
Token counts are useful proxies, not exact costs. Two prompts of equal length can differ because output duration, attention implementation, cache layout, sampling options, and model parallelism matter. A queue-delay predictor trained on yesterday's mix can fail after a product launch or model change. Keep a guard band, version thresholds with the model and serving configuration, and rerun load tests whenever those inputs change.
Isolation also spends capacity. A dedicated interactive pool may sit partly idle while bulk work waits, and strict reservations can reject a request that would have completed. Treat that as an explicit reliability trade rather than a scheduling bug. Review rejected demand, successful latency, starvation, and utilization together. When measured demand justifies more or different GPUs, describe the required model, token mix, concurrency, latency objective, and locality rather than relying on a request-per-second headline alone.