Prefix Cache Hit Rate Is Not a Capacity Plan

A prefix-cache hit says that some prompt tokens found reusable key and value state. It does not say how many requests fit in HBM, how long those blocks remain resident, or how much decode state will accumulate. Treating the hit percentage as a utilization forecast collapses two different resources into one number: prefill compute avoided and KV memory occupied. A fleet can report a strong hit rate while rejecting long generations because active sequences consume the remaining blocks.

This distinction follows from the mechanics. A transformer stores keys and values per layer for prior tokens, and the sequence dimension grows as context grows. PagedAttention reduces fragmentation by allocating fixed-size blocks and permits sharing, but it does not make those blocks free. Capacity still depends on model weights, runtime workspace, the KV representation, token residency, concurrency, and the scheduler's admission policy. Prefix reuse changes some terms, not the whole equation.

The compute ledger counts prompt tokens whose prefill work was avoided. For a measurement interval, define reusable prompt tokens as the matched prefix tokens actually served from cache, not tokens that merely looked identical in an application log. Divide those by all eligible prompt tokens to obtain a token-weighted reuse rate. Then segment it by tenant, route, model revision, adapter, and prompt-template version. A request-weighted rate can hide the fact that many tiny prompts hit while a few large prompts miss.

The residency ledger estimates KV token-seconds, not hits. For every request, track uncached prompt tokens, shared blocks referenced by that request, generated tokens, request lifetime, and the peak number of allocated blocks. Shared prefix blocks may have one physical copy but several references; private suffix and decode blocks generally grow with each live sequence. Report peak KV occupancy and eviction or recomputation pressure beside prefill savings. Neither metric substitutes for the other.

For a conventional attention layer, a useful planning identity is KV bytes per token = 2 times layer count times KV-head count times head dimension times bytes per stored element. The factor of two represents keys and values. Multiply by resident tokens, then account for tensor parallel placement and implementation-specific padding or metadata. Models using grouped-query attention have fewer KV heads than query heads, so using the total attention-head count can materially overstate memory. Confirm every field against the deployed model and engine.

Do not derive this value from parameter count or advertised context length. Measure the engine's allocatable KV pool after weights, graph captures, communication buffers, and runtime workspace are loaded. Convert that pool into blocks and tokens using the actual cache dtype and block size. Leave an operating reserve for allocation bursts and measurement error. The result is a hard residency envelope. Prefix behavior can improve how effectively requests share it, but cannot expand it.

Mixed tenants create separate reuse populations. A stable 2,000-token system prompt used every second is more likely to remain warm than fifty 2,000-token prompts each used once an hour, even though both collections could produce the same retrospective hit ratio during a favorable window. Prompt edits also change downstream block identities. vLLM's documented design hashes a block together with its preceding prefix, so a token change near the front can invalidate matches for all later blocks.

Build a reuse matrix with tenant or approved sharing domain on one axis and prompt-template version on the other. For each cell, record request rate, reusable block count, inter-arrival distribution, and estimated reuse distance in distinct blocks touched between uses. Compare that distance with the evictable cache budget. This is a stack-distance approximation, not a guarantee, because active requests pin blocks and scheduling order changes eviction. It is still more actionable than a fleet-wide daily hit rate.

Consider an illustrative pool holding 120,000 KV tokens after weights and reserve. Actual capacity varies by model, cache dtype, parallel layout, and engine. During a peak minute, Tenant A has 40 concurrent requests, each referencing one shared 2,000-token prefix and carrying a 500-token private prompt suffix plus a 1,000-token generated tail. Its residency estimate is 2,000 shared + 40 x (500 + 1,000) private = 62,000 tokens. Tenant B has 20 concurrent requests with unique 1,500-token prompts and 1,000-token outputs, using 20 x (1,500 + 1,000) = 50,000 tokens. Total estimated residency is 112,000 tokens, or 93.3 percent of the pool.

Now suppose Tenant A's 2,000-token prefix accounts for 80,000 matched tokens across the minute. Tenant A also contributes 40 x 500 = 20,000 uncached suffix tokens, while Tenant B contributes 30,000 prompt tokens that miss. The token-weighted hit rate is therefore 80,000 / (80,000 + 20,000 + 30,000) = 61.5 percent. The denominator includes every prompt token, including Tenant A's uncached suffixes. Yet only 8,000 resident-token slots remain, enough for roughly five more of Tenant A's 1,500-token private prompt-plus-output paths before block rounding. Longer outputs or a burst from Tenant B can exhaust the pool even if measured reuse improves. The cache saved prefill work and shared 78,000 duplicate prefix-token copies, but active private state still sets the immediate limit.

First, validate metric semantics: determine whether queries and hits count tokens, blocks, or requests, and reset or window counters consistently. Second, split time to first token into hit-depth bands while charting prompt throughput. Third, chart KV occupancy, waiting requests, preemptions or recomputations, and eviction activity against concurrent input plus output tokens. Fourth, repeat the view by tenant and template version. Fifth, replay a representative trace with caching enabled and disabled under the same arrival process, output caps, and service-level objectives.

Adopt a measurable decision rule. Add cache residency only if, across the peak-hour trace, the p95 time to first token misses its target, cached blocks are being evicted before their next reuse, and decode service remains within target. Add serving capacity or reduce admitted token concurrency if KV occupancy exceeds 85 percent for five consecutive minutes while the queue grows, regardless of hit rate. The 85 percent threshold is an illustrative operating rule, not a universal constant; calibrate it with failure tests and block-allocation behavior. ClusterBid can then match GPU memory configurations to the measured KV envelope rather than to the cache percentage alone.

Prefix caching only helps exact reusable prefixes at the engine's matching granularity. NVIDIA's TensorRT-LLM documentation notes that only full blocks are shareable in the described implementation, that reusable state can be evicted when active work needs memory, and that a second request may miss if it starts before the first finishes. Larger batches and longer outputs therefore can lower reuse opportunity while simultaneously raising memory pressure. Block size itself trades matching granularity against kernel efficiency, so it should be benchmarked rather than selected from hit rate alone.

Treat tenant isolation as a correctness and security constraint before treating cross-request sharing as an optimization. Adapter identity, model revision, multimodal inputs, prompt embeddings, tokenizer changes, and cache-salt or namespace policy may divide the cacheable population. Trace replay also has limits: it reproduces recorded demand, not the next product launch or a new prompt template. Keep a no-cache baseline, version the workload sample, and rerun the capacity calculation after changes to context limits, output caps, model architecture, cache dtype, batching policy, or tenant mix.

Automatic Prefix Caching, vLLM v0.8.5 documentation

KV cache reuse, TensorRT-LLM 0.18.2 documentation

Efficient Memory Management for Large Language Model Serving with PagedAttention

How caching works, Hugging Face Transformers documentation

FP4 Inference on Blackwell Cost Guide 2026: 8x Claims vs Production Reality

A/B Testing for AI Models: Production Inference Experimentation Framework