AI Inference Caching: KV Cache & Semantic Caching Strategies

LLM inference is memory-bandwidth bound, not compute bound, for most production serving configurations. The bottleneck is not the forward pass through the Transformer layers - it is the autoregressive token generation that forces repeated memory reads of the KV cache for every new token. A Qwen 2.5 72B model with 4k sequence length on an H100 SXM5 spends roughly 75% of inference time on attention operations dominated by KV cache reads. Caching strategies address this at two levels: reusing KV computations across requests with shared prefixes, and avoiding redundant LLM calls entirely through semantic result caching.

The cost savings translate directly to GPU hours. If your serving stack processes 10 million tokens per hour across your cluster and caching reduces effective compute by 35%, that is 3.5 million tokens of GPU time reclaimed per hour. At H100 on-demand rates of $1.15/hr per GPU on ClusterBid, a 35% cache hit rate on an 8-GPU serving deployment saves roughly $3,200 per month. For teams serving hundreds of thousands of requests daily, caching is not an optimization - it is the difference between positive unit economics and negative unit economics for inference.

There are three caching tiers worth implementing in any production inference stack: prefix KV cache reuse within a single LLM engine instance, semantic result caching that returns cached responses for semantically equivalent queries at the application layer, and cross-instance distributed cache sharing using a backing store like Redis or Memcached. Each tier addresses a different failure mode in the caching hierarchy, and each has specific cost and latency tradeoffs that determine where it fits in your serving architecture.

KV cache reuse exploits a simple property of autoregressive Transformers: if two inference requests share a common initial sequence of tokens, the KV pairs for that prefix are identical and can be computed once and reused. This is most impactful for system prompts, instruction prefixes, chat templates, and few-shot examples that appear in every request. A chatbot serving a fixed system prompt of 500 tokens that receives 100,000 requests per day is computing the same 500 tokens of KV cache 100,000 times. Caching that prefix eliminates 50 million token positions of redundant computation per day.

vLLM implements prefix-aware KV cache management through its automatic prefix caching feature. When enabled, vLLM's PagedAttention stores KV blocks in a hash table indexed by the hash of their token sequence. Requests arriving with tokens matching a cached prefix automatically reuse the cached KV blocks, only computing new KV pairs for the suffix. The cache eviction follows an LRU policy within each GPU's memory budget. This approach adds near-zero overhead to the serving path - the hash lookup completes in microseconds and the cache reuse requires no inter-GPU coordination because each GPU manages its own prefix cache independently.

The memory impact of prefix caching is significant. For a 72B model at FP16 with 4k sequence length on an 8x H100 node, the KV cache consumes approximately 30GB of HBM per GPU without prefix reuse. A 500-token prefix that is fully cached saves roughly 4GB per GPU of KV cache memory. With typical serving batch sizes of 16-32 concurrent requests, the memory savings compound - you can serve more concurrent requests per GPU before hitting memory capacity limits. This directly increases throughput per dollar for your serving cluster.

Semantic caching extends the caching idea beyond exact token sequence matches. Instead of requiring identical prefixes, semantic caching uses embedding similarity to detect when a user query is asking the same question in different words. Two queries like 'What is the refund policy?' and 'Can I get my money back?' have different token sequences but express the same intent. A semantic cache can return the cached response for the first query when the second arrives, bypassing the LLM entirely.

Implementation requires an embedding model - typically Sentence-BERT, E5, or the embedding endpoint of your LLM provider. Each incoming query is embedded into a vector, and the cache stores query-response pairs keyed by their embedding vectors. A vector database (Milvus, Qdrant, PGVector, or a simple FAISS index) performs an approximate nearest neighbor search against the embedding of the current query. If the nearest neighbor has cosine similarity above a configurable threshold (typically 0.92-0.97), the cached response is returned. The embedding computation adds roughly 5-15ms to request latency, which is negligible compared to the 500-5000ms saved by avoiding the LLM call.

The threshold selection is critical. Too low (e.g. 0.85) and you risk returning semantically inappropriate cached responses for queries that are similar but have different correct answers. Too high (e.g. 0.99) and you approach exact match behavior, defeating the purpose of semantic caching. The right threshold depends on the diversity of your query distribution and the cost of serving an incorrect cached response. For customer support applications where wrong answers have high reputational cost, start at 0.97 and tune downward based on measured cache hit rates and error rates. For documentation Q&A where answers are well-defined, 0.92-0.95 is appropriate.

The most effective caching architecture for production inference is a three-tier hybrid cache. Tier one is the prefix KV cache inside the LLM engine (vLLM, TensorRT-LLM), operating at the GPU memory level with zero additional latency. Tier two is an exact-match application cache (Redis or Memcached) keyed on the input token hash, which catches identical requests that bypass prefix caching because they are processed by different engine instances. Tier three is the semantic cache using embedding similarity, which catches the largest fraction of redundant queries but adds the most latency.

The routing logic for the hybrid cache: incoming request checks the exact-match cache first (sub-millisecond Redis lookup). On miss, it checks the semantic cache (5-15ms for embedding and ANN search). On semantic miss, the request proceeds to the LLM engine, where vLLM's prefix caching may still provide partial reuse. The response is then written back to both the exact-match cache and the semantic cache with its embedding for future requests. This layered approach achieves cumulative hit rates of 50-70% depending on your query diversity, compared to 20-40% from prefix caching alone.

Cache invalidation is the hardest problem. For prefix KV caches, invalidation is handled automatically by LRU eviction - stale entries are evicted as new prefixes fill GPU memory. For semantic caches, you need a TTL-based invalidation policy or an active invalidation trigger when the underlying knowledge changes. If you update your system prompt or retrain your model, the semantic cache may return responses that reference outdated model behavior. The safest approach: set a TTL of 24 hours for semantic cache entries and actively invalidate entries matching specific query patterns when you deploy model or prompt changes.

When you scale beyond a single GPU or node, prefix caching becomes fragmented. Each GPU manages its own prefix cache independently, so a request hitting GPU B cannot reuse the prefix computed on GPU A. For clusters serving a common set of prompts across multiple replicas, this fragmentation wastes significant compute. The solution is a distributed KV cache that pools prefix cache state across all GPUs in the serving cluster, using a shared backing store that any GPU can read from.

The practical implementation uses Redis or Memcached to store serialized KV cache blocks indexed by the token prefix hash. When a request arrives at any GPU, the engine checks the distributed cache for the prefix before computing any new KV pairs. If the prefix exists, it deserializes the cached blocks into the local PagedAttention manager. If not, it computes the prefix and writes a copy to the distributed cache for future use. The overhead of serialization and deserialization is roughly 1-3ms for a 500-token prefix, compared to the 50-150ms saved on redundant prefix computation.

The distributed cache requires fast networking between the GPU nodes and the cache store. A Redis cluster running on the same InfiniBand fabric as your GPU nodes reduces serialization latency to 200-400 microseconds for KV block transfers. Using cloud-hosted Redis adds 5-20ms of network latency that can negate the caching benefit for short prefixes. For distributed cache deployments, co-locate the Redis or Memcached instances on the same cluster network as your GPUs, ideally within the same rack. The networking overhead of cross-rack cache reads in a multi-rack cluster can exceed the prefix computation time for prefixes under 200 tokens.

Caching infrastructure has its own costs: the embedding model inference for semantic caching (a small model like E5-mistral-7b can run on a fraction of an H100 or a cheap CPU instance), the vector database or Redis cluster, and the engineering time to implement the caching layer. A full hybrid cache deployment for a production serving cluster costs roughly $300-800 per month in auxiliary compute and storage, plus initial engineering investment of 2-4 weeks.

The payback period is typically 2-6 weeks depending on your inference volume. For a cluster serving 5M tokens per day from a 70B model at $1.15/hr per H100 GPU, the daily inference cost is roughly $350 at $0.37 per million tokens. A 50% cache hit rate reduces effective token volume to 2.5M tokens per day, saving $175 per day - roughly $5,250 per month. The caching infrastructure cost of $500 per month is recovered in under 3 days. The ROI equation changes only for very low-volume serving deployments under 100k tokens per day, where caching infrastructure cost may exceed the savings.

Beyond direct GPU cost savings, caching reduces inference latency significantly. A cached semantic response returns in 100-300ms compared to 2-8 seconds for a full LLM generation. For user-facing applications where every 100ms of latency reduces conversion rates by 1-2%, the indirect revenue impact of caching often exceeds the direct compute savings. This is the argument that gets executive buy-in for caching investment: faster responses drive higher user engagement, and caching delivers that at negative net infrastructure cost.

Start with vLLM automatic prefix caching. Enable it with a single flag (--enable-prefix-caching) and measure your baseline hit rate on production traffic before building any additional caching layers. Most teams see 15-30% cache hit rate from prefix caching alone, and it costs nothing to enable. Monitor the hit rate per GPU and verify that prefix caching is not increasing TTFT (time to first token) - in some pathological cases with very diverse prefixes, the hash table overhead can add latency without meaningful cache reuse.

Add the exact-match Redis cache next. This is the lowest engineering effort for the highest marginal gain above prefix caching. The implementation is a simple interceptor that checks a SHA256 hash of the full input against Redis before routing to the LLM. Most web frameworks have middleware or interceptor patterns that make this a 50-line code change. Set a TTL of 1 hour initially and adjust based on your query repetition patterns.

Finally, add semantic caching with an embedding model sized to fit your latency budget. For sub-50ms embedding latency, use a distilled Sentence-BERT model that runs on CPU (int8 quantized all-MiniLM-L6-v2). For higher accuracy, run a larger embedding model on GPU - but allocate at most 5% of a single H100 GPU to the embedding service to keep the infrastructure cost proportionate. Deploy the semantic cache with a conservative similarity threshold (0.97) and gradually lower it while monitoring cache correctness through spot-check audits of cached versus fresh responses. Implement the full three-tier cache before optimizing further - the engineering time spent on niche cache optimization before the basics are in place is nearly always wasted effort.