Why a Model Fits in VRAM but Still Fails Under Load

A model that reaches a ready state has shown that its weights and startup allocations fit once. It has not shown that the server can admit the intended mix of requests. During service, memory also holds key-value cache blocks, temporary activations, attention and GEMM workspaces, communication buffers, allocator reservations, and possibly captured CUDA graphs. Concurrency and sequence length make several of those terms grow after startup.

Write the budget as VRAM capacity minus weights, persistent runtime allocations, peak workspace, graph pools, KV cache, and a safety reserve. Measure allocated and reserved memory separately. A caching allocator may retain freed blocks, so a monitoring chart can show high reserved memory without the same amount being live tensors. Conversely, a brief workspace peak can cause an allocation failure between coarse monitoring samples.

Take measurements at the same lifecycle points on every build: after process start, after weight load, after representative warmup, and at the busiest sustained interval. Device-wide readings catch memory held outside the framework, while allocator counters explain the framework's share. Reconcile the two rather than selecting whichever number looks more favorable. A widening gap can indicate library, graph, communication, or driver allocations that the tensor ledger does not include.

The first estimate is parameter count multiplied by stored bytes per parameter, adjusted for quantization metadata, scales, duplicated tensors, and sharding. An illustrative 13 billion parameter model stored at 2 bytes per parameter starts at 13,000,000,000 x 2 = 26,000,000,000 bytes, or about 24.2 GiB after dividing by 1,073,741,824. That is not a promise that its process consumes 24.2 GiB. File format, tied weights, kernels, and runtime representation matter.

Confirm the estimate after loading with framework and device measurements. On a sharded replica, inspect every rank because layers and auxiliary allocations may not divide perfectly. Keep engine artifacts and process count in the inventory. Two worker processes on one GPU can each load weights or establish their own memory pools, turning a configuration mistake into what looks like random load-dependent OOM.

Quantization makes file size an especially poor proxy. Scales, zero points, packing alignment, and layers retained at a wider precision all consume space. Some runtimes also deserialize into a representation that differs from the checkpoint. Record measured resident bytes for the exact engine artifact and software versions, then rerun the load measurement after rebuilding or changing quantization settings.

KV cache grows with tokens retained across active sequences. For an unsharded conventional decoder estimate, bytes per token for one sequence are approximately 2 x layers x KV heads x head dimension x bytes per element. The factor two represents keys and values. Architectures with grouped-query or multi-query attention use fewer KV heads than query heads, and tensor parallel engines may divide KV heads or cache blocks across ranks, so use the deployed model configuration and engine layout rather than substituting the total attention-head count.

For an illustrative configuration with 32 layers, 8 KV heads, head dimension 128, and 2-byte cache elements, one token needs 2 x 32 x 8 x 128 x 2 = 131,072 bytes, or 128 KiB. Ten concurrent sequences retaining 8,000 tokens each require about 10 x 8,000 x 128 KiB = 9.77 GiB before block-allocation waste. Actual implementations can add alignment, metadata, or alternate cache formats. Prefix reuse changes physical demand but should be measured, not assumed.

Libraries choose algorithms that may require temporary workspace. TensorRT documents its builder workspace limit as a control that can exclude tactics needing more scratch memory; it is not a cap on every runtime allocation. At runtime, execution contexts and CUDA graph captures can retain their own memory resources. Warmup with one short request may not activate the largest shape, batch, or graph bucket. Exercise the maximum admitted prompt length, decode length, batch composition, and parallel mode during qualification, then record the high-water mark.

Graph capture trades launch overhead for persistent memory and shape constraints. Capturing many batch or sequence buckets can improve coverage while consuming more memory. The decision rule is measurable: retain a capture bucket only if its verified latency or throughput gain is worth its observed incremental VRAM under the production mix. Otherwise reduce captured shapes or use eager execution for uncommon shapes. Recheck after engine rebuilds because tactic and workspace choices can change.

Measure peaks with synchronization around the region under test because asynchronous GPU work can make host-side timing misleading. Run each admitted shape more than once and watch for one-time allocations during the first execution. Then repeat the sequence in production order. A set of shapes that passes individually can still fail when cached plans or captured graphs remain resident together.

Consider an illustrative device with 80 GiB of usable capacity. Suppose measured weights consume 24.5 GiB, persistent runtime and communication allocations 3.0 GiB, peak workspace 4.0 GiB, graph pools 5.0 GiB, and the operator reserves 6.0 GiB. These categories must be measured as disjoint allocations that can coexist; otherwise summing independent peaks double counts reused memory. Under that assumption, the KV allowance is 80 - 24.5 - 3 - 4 - 5 - 6 = 37.5 GiB. At the illustrative 128 KiB per token, that holds 307,200 token slots because 37.5 GiB divided by 128 KiB equals 307,200.

If the measured workload's p99 retained context is 6,000 tokens per active request, a naive ceiling is 307,200 / 6,000 = 51 concurrent requests. Do not configure 51 immediately. Block granularity, fragmentation, prompt bursts, output uncertainty, and per-request state lower the safe value. Sweep concurrency using the real length distribution and select the highest level that produces zero OOM or preemption events while meeting latency over a sustained soak. Every number here is illustrative.

First identify whether failure occurs at load, warmup, prefill, decode, or a concurrency transition. Second capture per-rank free, allocated, and reserved memory plus active sequence tokens. Third compare the failing request's input and requested output limits with admitted policy. Fourth check worker count and graph captures. Fifth reproduce with one variable changed: concurrency, context limit, graph mode, or workspace setting. An OOM during long prefill suggests a different lever from gradual KV exhaustion during decode.

A practical gate is to reject or queue work when projected KV blocks plus observed non-KV high-water usage would cross the safety reserve. Also alert on preemptions before OOM because they can damage latency while the process remains alive. Limitations remain: allocator fragmentation is workload-dependent, unified-memory behavior can obscure device pressure, and estimation formulas differ by architecture and cache precision. The final authority is a soak test using the deployed engine and request distribution. Use that measured ledger when matching ClusterBid capacity to the workload.

TensorRT 10.4 Developer Guide

vLLM 0.6.2 Documentation

CUDA Graph memory management

PyTorch CUDA memory management

FP4 Inference on Blackwell Cost Guide 2026: 8x Claims vs Production Reality

A/B Testing for AI Models: Production Inference Experimentation Framework