Prefix reuse begins with exact token identity
A shared system prompt helps a prefix cache only when requests begin with the same token sequence under the same cache identity. Similar wording is not enough. Whitespace, role markers, tool definitions, JSON ordering, timestamps, locale text, and template separators can change tokens before the user message begins. Once an early block differs, later matching prose does not restore the broken prefix chain for a hash-based cache.
Inspect the rendered prompt and token IDs at the serving boundary. Application-level message objects are too early because a chat template can add control tokens and rearrange fields. Store a non-sensitive fingerprint of the tokenizer revision, template revision, model revision, and leading token blocks with each observation. That makes a hit-rate change attributable without logging private prompt text.
Understand the cache unit before changing templates
The vLLM 0.7.3 prefix-caching design identifies a KV block from its tokens and the preceding prefix, with additional identity components where needed. That design caches only full blocks. Therefore, with a 16-token block size, a request sharing 70 leading tokens with a prior prompt can reuse at most 64 tokens, assuming those four complete blocks are still cached and all other cache identity inputs match. The six-token partial block is computed again. Block granularity explains why a small token edit can cause a larger step in reused-token counts.
Prefix caching reduces repeated prefill computation. It does not eliminate decode cost, and it does not guarantee lower end-to-end latency when queueing or generation dominates. Evaluate cached prompt tokens and time to first token by prompt-length band. Overall cache hit rate can rise while user latency stays flat if the hits cover short prefixes or occur on requests dominated by long outputs.
Treat template edits as cache-key changes
Changing a system prompt at the first token invalidates reuse with entries built from the old version, even if most wording remains. Appending stable guidance near the end preserves earlier matching blocks but still changes later ones. Inserting a request-specific timestamp or identifier near the front destroys reuse for everything after it. Put stable, broadly shared content first and variable content later, provided that ordering remains semantically correct for the model.
Version the rendered template, not just the human-readable system text. A tool schema serializer, escaping rule, or role wrapper can alter the byte string and token sequence without a content editor noticing. During deployment, compare leading token IDs for a fixed fixture across old and new revisions. If they differ, predict a cold-cache period and capacity impact rather than treating a cache-hit collapse as an incident after release.
| Change | Likely reuse effect | Check |
|---|---|---|
| Edit first system sentence | Invalidates following prefix chain | Leading token diff |
| Append stable instruction | Earlier full blocks may still match | Matched block count |
| Insert timestamp near front | Fragments reuse per request | Cardinality of leading fingerprints |
| Change tokenizer or chat template | Token IDs or control tokens may change | Pinned fixture tokenization |
| Change tenant salt | Separates otherwise equal prefixes | Salt scope and hit metrics |
Pin tokenizer, model, and cache identity together
The same text need not yield the same token IDs after a tokenizer configuration or vocabulary changes. Chat templates are also part of tokenization behavior. Pin model artifacts, tokenizer files, added-token configuration, normalization behavior, and template revision as one serving release. Run golden fixtures that compare the complete rendered prompt and token ID sequence, including beginning and end markers.
Do not share KV state across model weights or adapters merely because input token IDs match. Cached keys and values are products of the model computation, so cache identity must include the model state and relevant adapter identity. The vLLM design notes extra hashes such as LoRA identifiers. When deploying changed weights, start a separate cache namespace or reset the appropriate cache according to the engine contract.
Make cache sharing follow trust boundaries
Cross-request reuse can expose a timing signal because a cached prefix may reach the first token faster than a cold one. Do not treat prefix caching as an isolation boundary. Separate serving processes provide a stronger boundary than cache namespacing. Where the pinned engine version supports it, a gateway can instead attach an unpredictable cache salt scoped to an authenticated tenant or approved trust group. Do not let callers choose another tenant's salt, and do not use a guessable tenant name as the secret.
The vLLM 0.17.0 design describes injecting cache_salt into the first block hash so only requests with the same salt can reuse following blocks. Salting limits this cache timing channel, but it does not make a shared process a complete tenant-isolation mechanism. Keep authorization, quotas, logs, model access, and data separation independent. Measure hits by a bounded internal namespace label rather than raw customer identifiers, and verify salt handling in the exact API and engine version deployed.
Worked example: estimate a template rollout
Assume an illustrative warmed interval contains 100 requests. Tenant A sends 80 requests, each with a 1,024-token shared prefix and a 500-token uncached suffix. Tenant B sends 20 prompts of 600 tokens with no matching prefix. With a 16-token block size, Tenant A's prefix is 64 complete blocks. If measurements show that 75 percent of those prefix tokens were served from cache, cached tokens equal 80 multiplied by 1,024 multiplied by 0.75, or 61,440 tokens.
For a prompt-token cache hit rate, the denominator must include every prompt token in the measured population: Tenant A's matched prefixes, Tenant A's 500-token uncached suffixes, and Tenant B's prompts. The denominator is 80 multiplied by 1,524 plus 20 multiplied by 600, or 133,920 tokens. The overall hit rate is therefore 61,440 divided by 133,920, or about 45.9 percent, not 75 percent. Seventy-five percent is only the effectiveness rate within Tenant A's eligible shared prefixes. All quantities are hypothetical, and neither percentage is a latency or capacity claim.
Measure invalidation with controlled fixtures
Track total prompt tokens, eligible shared-prefix tokens, cached prompt tokens, time to first token, eviction activity, and cache occupancy by template revision. Report two ratios with explicit denominators: cached tokens divided by all prompt tokens for population-wide hit rate, and cached tokens divided by eligible shared-prefix tokens for reuse effectiveness. Do not label either as merely hit rate, and do not use requests with any hit as the denominator. Run three fixtures: identical prefix, one-token early change, and one-token late change. Their reused-token counts should follow the engine's block rules and reveal accidental normalization or template drift.
There is rarely a need to flush every cached block for an ordinary prompt-text edit because new token hashes naturally miss old entries and eviction can reclaim them. Explicit invalidation is appropriate when weights or cache identity semantics change, when testing requires a cold baseline, or when an engine documents a correctness reason. A cache reset can cause a load spike, so schedule it with headroom. Prefix-cache metrics are implementation-specific; validate names and semantics against the pinned engine version before building an alert.