Sizing Host RAM for GPU Inference Workers

GPU memory charts do not describe the health of an inference worker's host. CPU RAM holds checkpoint pages during loading, tokenizer state, request bodies, queues, application caches, process heaps, file-system cache, observability agents, and page-locked transfer buffers. A service can have abundant free VRAM and still be killed by a Linux cgroup or the host OOM killer.

Start with a ledger for the whole node and another for each container. Include the operating system and node services before multiplying worker costs. Distinguish resident set size, cgroup current usage, mapped files, page cache, and pinned memory where the tooling permits. Summing process RSS can double-count shared pages, while looking only at one process can miss sidecars and child workers.

Define the failure boundary as well as the capacity. A Kubernetes memory limit, a systemd unit limit, and physical RAM are different ceilings. Record which one the worker can reach first and reserve memory outside it for node stability. Huge pages, shared memory mounts, and tmpfs can also draw from host memory. Inventory them explicitly instead of assuming that only anonymous process memory matters.

Naive loading can temporarily hold initialized model weights plus checkpoint weights in host memory. Transformers 4.19 documents low_cpu_mem_usage, which discards the randomly initialized weights and progressively loads pretrained weights. The same version says its described sharded-checkpoint workflow caps CPU RAM at the model size plus the largest shard. These are properties of that loader path, not universal guarantees for every engine or checkpoint format, so verify the exact version and format you deploy.

Measure from process start through readiness and the first representative requests. Rolling replacement can overlap an old worker and a new worker on one node, so the node peak may be two serving processes plus one loader rather than a single steady process. If the orchestrator permits surge, include it explicitly or configure placement so replacements do not overlap on a RAM-constrained node.

Tokenization and request assembly scale with active requests and input size. Response buffers scale with generated text when clients are slow. Prefix, prompt, or application caches scale with their configured limits, not merely current occupancy. Each process adds interpreter, libraries, thread stacks, allocator arenas, and metrics state. Forking may share clean pages initially, but copy-on-write can erode that saving.

Pinned host buffers deserve their own line. NVIDIA's CUDA best-practices guide states that page-locked memory enables high-bandwidth and asynchronous transfers, but warns that excessive use can reduce system performance because it is scarce. Pool and cap staging buffers rather than allocating without bounds per request. Do not count swap as normal inference capacity; reclaim and swap stalls can violate latency long before an OOM kill.

Consider an illustrative 256 GiB node. Reserve 24 GiB for the OS, filesystem activity, drivers, and agents, plus 32 GiB as a pressure reserve. Suppose four steady workers each measure 34 GiB at the chosen load, contributing 4 x 34 = 136 GiB. Shared application caches are capped at 20 GiB and pinned staging pools at 8 GiB. Total planned usage is 24 + 32 + 136 + 20 + 8 = 220 GiB, leaving 36 GiB unassigned.

Now measure startup. If one replacement worker peaks 18 GiB above its steady 34 GiB while the old worker remains, planned usage becomes 220 + 18 + 34 = 272 GiB, which exceeds 256 GiB by 16 GiB. The safe choices are no-surge replacement, lower worker count, a lower-memory loader, or more host RAM. All figures are illustrative planning assumptions. Actual memory and hardware costs vary; use observed high-water marks.

First check whether the process exited, was evicted, or returned a CUDA allocation error. Second inspect container status and the kernel log for OOM-kill evidence. Third read cgroup v2 memory.events, especially oom and oom_kill, and compare memory.current with memory.max. Fourth inspect memory.stat and pressure stall information. Fifth align those events with worker RSS, cache occupancy, request count, and loading activity.

A host OOM kill may leave no Python exception because the kernel terminates the process. A CUDA OOM normally originates from a device allocation and should be correlated with device memory and engine logs. Do not infer one from the other. A practical alert rule is any increase in cgroup oom or oom_kill counters, or sustained memory pressure combined with current usage above 90 percent of the cgroup limit. The 90 percent threshold is illustrative and should be tuned to reclaim behavior.

Run a sustained replay at target concurrency, then exercise model load, rolling replacement, cache fill, slow clients, telemetry export, and a worker restart. Accept the node only if it stays below its reserve, records no OOM events, and meets latency objectives. Track high-water usage by deployment version. When fitting ClusterBid capacity to requirements, specify host RAM per GPU, process topology, loader peak, and rollout policy, not just accelerator memory.

Host sizing remains environment-specific. Memory-mapped checkpoint pages may be reclaimable, shared libraries complicate attribution, allocator fragmentation changes over time, and page cache is useful until pressure forces reclaim. Container limits can fail below physical node capacity, while an unconstrained container can harm neighbors. Recalculate after changing model format, worker count, tokenizer parallelism, cache limits, staging strategy, sidecars, or rollout settings.

Hugging Face Transformers v4.19 Big Models

CUDA C++ Best Practices Guide: Pinned Memory

Linux kernel Control Group v2 documentation

FP4 Inference on Blackwell Cost Guide 2026: 8x Claims vs Production Reality

A/B Testing for AI Models: Production Inference Experimentation Framework