Prefill and decode stress the GPU differently
An LLM request is not a single homogeneous job. Prompt processing and token generation have different parallelism, memory traffic, and user-facing consequences.
Process prompt tokens in parallel
Build the first logits and write keys and values for every attention layer and live prompt token.
- Tendency
- Compute-bound for long prompts or useful batches
- User sees
- Time to first token
- Pressure
- Prompt length, attention cost, queueing
Generate one token per sequence per step
Read weights and prior KV, append one entry, sample, then repeat. Each response is sequential even when many responses batch together.
- Tendency
- HBM-bandwidth-bound at low-to-medium batch
- User sees
- Inter-token latency / TPOT
- Pressure
- Live sequences, output length, KV occupancy
A single tokens-per-second number hides the service
Define where the clock starts, publish tail percentiles, and keep input, output, request, and total-token throughput separate.
| Metric | Definition | What it diagnoses |
|---|---|---|
| TTFT | Request submission to first output token | Gateway, queue, tokenization, prefill, first decode, cold state |
| ITL | Each interval between adjacent streamed tokens | Visible stalls and tail cadence |
| TPOT | Average post-first-token time per generated token | Average decode cadence per request |
| E2E latency | Submission to final token | Whole response experience |
| Output throughput | Generated output tokens / wall time | Fleet generation capacity |
| Request throughput | Completed requests / second | Only comparable with the same length distributions |
| Goodput | Requests or tokens completed inside latency, quality, and error SLOs | Capacity that is safe to sell or plan |
E2E ≈ TTFT + (O − 1) × TPOT- Report p50/p95/p99 TTFT and ITL or TPOT, not averages alone.
- Record prompt/output distributions, arrival rate, burstiness, concurrency, prefix reuse, and sampling settings.
- Keep engine-internal latency separate from user-observed gateway-to-token latency.
- Compare goodput only at the same model revision, tokenizer, precision, lengths, engine version, and SLO.
KV cache grows with live tokens, not parameter count
The model weights may fit while production concurrency does not. Plan keys and values for every live prompt and generated token.
KV bytes = 2 × layers × KV heads × head dimension × bytes × Σ live sequence tokens32 layers × 8 KV heads × 128 head dimension × 2 bytes × key/value = 128 KiB per live token.
One 8,192-token sequence is roughly 1 GiB of raw KV. Thirty-two such live sequences are roughly 32 GiB - before block rounding, runtime pools, and other overhead.
What reduces KV
GQA/MQA, fewer live tokens, sliding-window/hybrid attention, lower KV precision, prefix block sharing, and architecture-specific cache designs.
What multiplies KV
Concurrency, long outputs, beam search, n > 1, multimodal tokens, block rounding, and replicated KV heads at some tensor-parallel degrees.
Paged attention manages allocation, not physics. Block-based KV reduces fragmentation and enables sharing, but the actual K/V payload still consumes memory.
Prefix caching removes repeated prefill work. It helps TTFT when token-identical prefixes recur; it does not accelerate the uncached suffix or ordinary decode. Template changes, timestamps, whitespace, tokenizer versions, and round-robin routing can destroy reuse.
Batch continuously, then tune the tradeoff you actually own
| Lever | It helps when | It can hurt when | Track |
|---|---|---|---|
| Continuous batching | Sequences have varied lengths and traffic is concurrent | Admission is too aggressive for KV or tail SLO | Active/waiting sequences, batch tokens, queue, goodput |
| Chunked prefill | Long prompts interfere with streaming decode | Chunk/token budget adds scheduling overhead or worsens TTFT | TTFT, ITL, prefill chunks, queue |
| Prefix caching | Stable token-identical prefixes recur | Templates vary, routing fragments locality, cache churns | Query/hit tokens, eviction, warm vs cold TTFT |
| Weight quantization | Weights do not fit or decode is weight-bandwidth-bound | Kernel fallback/dequant or quality loss dominates | HBM, quality, prefill and decode separately |
| KV quantization/offload | Context/concurrency is the memory limit | Conversion or host transfer increases latency | KV capacity, occupancy, transfer, p99 ITL |
| Speculative decoding | Low/medium batch decode is bandwidth-bound and proposals are accepted | Large batches saturate target or proposer overhead is high | Accepted length, draft/verify time, TPOT and total throughput |
Capacity planning is trace replay plus memory feasibility
- Capture demand.Prompt/output distributions, arrival pattern, burstiness, prefixes, model mix, sampling and SLOs.
- Calculate fit.Subtract measured weights, workspace, graphs, and reserve from HBM; estimate the remaining physical KV-token capacity.
- Benchmark one request.Establish the best attainable TTFT and TPOT before queueing and batching.
- Sweep concurrency and arrival rate.Replay realistic and bursty traffic through the queueing knee, including long-prompt probes.
- Select the SLO operating point.Use the highest rate where all latency, quality, and error gates pass - not saturation throughput.
- Add headroom and failure capacity.Test a missing replica, rollout overlap, cold node, cache loss, and model reload.
replicas ≈ ceil(target demand ÷ goodput per replica × (1 + reserve fraction))output token demand/s = request rate × E[output tokens]