Treat one endpoint as three capacity problems
A multi-tenant LoRA endpoint does not have one simple memory budget. The base model consumes a largely fixed allocation on every replica. Resident adapters consume another allocation that changes with the active tenant set. Request state, especially the key-value cache for attention, consumes a third allocation that changes with prompt length, generated length, batch composition, and concurrency. Combining these into a single GPU utilization percentage hides which resource is actually limiting service.
Keep three ledgers. Record base weights and serving-engine overhead as the replica floor. Record adapter bytes by adapter, rank, target modules, and precision. Record KV-cache use in blocks or bytes alongside active tokens. A replica can have room for another adapter yet lack KV capacity for a long prompt, or have free KV blocks while repeatedly loading cold adapters. LoRA reduces per-tenant weight duplication, but it does not make tenant switching or request state free.
Separate the residency set from the working set
The catalog may contain thousands of adapters, while only a small subset must be on a GPU at once. S-LoRA explicitly separates these ideas by keeping adapters in main memory and fetching those used by running queries into GPU memory. Its unified paging design manages adapter weights of different ranks together with variable-sized KV tensors. That research result is an architectural option, not proof that every serving stack provides identical paging behavior or overhead.
Define the GPU residency set as adapters immediately callable without a host-to-device load. Define the host-resident set as adapters staged in CPU memory but absent from the GPU. Everything else is remote or disk-resident. Capacity planning should assign a byte limit and an expected load latency to each tier. Do not use adapter count alone: two adapters with different ranks or target-module coverage may have materially different sizes. Validate the serialized size and the in-memory representation used by the chosen runtime.
| Capacity pool | Primary driver | Failure signal | Control |
|---|---|---|---|
| Base model | Weights, precision, parallel layout | Replica cannot start or leaves too little headroom | Model format, quantization, sharding |
| Adapter residency | Adapter bytes and active tenant mix | Loads, evictions, and queue stalls rise | Placement, admission, cache size |
| KV cache | Active tokens and concurrency | Preemption, recomputation, or rejected work | Token budget, batching, replica count |
Measure churn at the request boundary
Instrument every request with adapter identity, replica identity, queue time, adapter lookup result, load start and end, prompt tokens, generated tokens, and completion status. From those events calculate GPU residency hit rate, host-tier hit rate, adapter loads per minute, evictions per minute, and load-time percentiles. Also track time to first token and inter-token latency by hit or miss class. A global latency percentile cannot reveal whether cold tenants are waiting behind loads while hot tenants remain healthy.
Use a diagnostic sequence when latency rises. First, check whether arrival rate or active-token count changed. Second, compare adapter miss and eviction rates with their prior baseline. Third, split time to first token into queue, adapter preparation, and prefill where the runtime exposes those phases. Fourth, inspect whether misses cluster on particular replicas, which suggests poor routing rather than insufficient fleet capacity. Finally, check KV pressure and preemption. If misses are flat but active tokens and preemptions rise, adding adapter slots is unlikely to fix the incident.
Work a residency budget before choosing hardware
Consider an illustrative replica with 80 GiB of device memory. Assume the base model and runtime reserve 48 GiB, the operator preserves 4 GiB of safety headroom, and the measured adapter representation averages 240 MiB. If 20 GiB is reserved for KV cache, 8 GiB remains for adapters. Using binary units, 8 GiB equals 8,192 MiB, so the nominal adapter capacity is floor(8,192 / 240) = 34 adapters. This is an illustrative planning calculation, not a hardware benchmark, and actual memory use varies by model, runtime, rank, precision, and allocator behavior.
Now assume an observed ten-minute window contains 50,000 adapter-bearing requests and 46,500 find their requested adapter already resident at admission. The measured GPU-residency hit rate is 46,500 / 50,000 = 93 percent. Exclude requests that use no adapter, or report them separately, so the denominator measures the lookup population it claims to describe. If the service objective requires at least 97 percent of adapter-bearing requests to avoid a GPU adapter load, this layout fails even if average GPU utilization looks comfortable. Do not immediately take memory from KV cache. Replay or canary-test a larger adapter pool and verify that KV preemption and long-context latency remain within their budgets. Actual infrastructure prices vary, so this decision should use measured workload behavior rather than an assumed dollar rate.
Route for affinity without trapping capacity
Adapter-aware routing sends a request to a replica where its adapter is resident, provided that replica has acceptable queue and KV headroom. This preserves warm state and reduces transfers. Strict affinity can backfire when a popular tenant overloads one replica while others idle. Use bounded affinity: prefer a warm replica only while its predicted queue delay stays below the cold-load penalty on a less busy replica. The comparison must use observed load time and queue time, not a permanent tenant-to-GPU mapping.
Batch formation matters after placement. Punica demonstrates that operations for different LoRA models can be batched with specialized kernels, so homogeneous adapter batches are not the only viable design. Still, runtime support and performance vary. Track tokens processed per scheduling step and batch diversity, then test whether admitting another distinct adapter reduces useful throughput or increases latency. A practical scheduler may reserve a few active slots for high-frequency adapters while leaving at least one slot available for the long tail, preventing a pinned hot set from blocking every cold tenant.
Turn cache observations into admission rules
Adopt a measurable rule for residency changes. One useful rule is: expand or repartition the adapter pool only when the seven-day peak-hour residency hit rate is below its target for three consecutive windows, adapter preparation contributes more than the allowed share of time to first token, and KV preemption remains below its own threshold. All three conditions matter. A low hit rate with tiny load cost may not justify memory, while high load delay during severe KV pressure calls for another replica or tighter token admission rather than a larger adapter cache.
For tenant admission, estimate each adapter's request frequency, measured bytes, and miss penalty. Prefer residency for adapters with the greatest avoided waiting time per MiB, recalculated on a rolling window. Put bursty tenants behind a concurrency limit and prewarm only when a credible scheduled event justifies it. When matching capacity to these measured requirements through ClusterBid, compare candidate configurations by usable memory after the base-model floor, not by headline GPU memory alone. Preserve the same canary trace across candidates so placement policy is not confused with hardware capacity.
Know what cache metrics cannot prove
A cache hit is not automatically fast. Activation bookkeeping, kernel selection, rank heterogeneity, tensor-parallel communication, or a crowded batch can still increase latency. Conversely, an adapter miss served from pinned host memory may be tolerable at low load. Research systems such as S-LoRA and Punica report strong results under their evaluated models, hardware, kernels, and workloads, but those figures should not be copied into a capacity forecast. Reproduce the relevant request mix on the actual serving version and treat upgrades as fresh measurements.
Run the operating loop weekly and after every model, adapter, or runtime change. Recompute adapter sizes, residency and host-tier hit rates, eviction rate, load-time percentiles, active tokens, KV preemptions, and latency by tenant. Review the coldest and hottest adapters, then adjust routing, pins, admission, or replica count one variable at a time. Roll back when the target metric improves but a guardrail regresses. This discipline keeps base-model capacity, adapter residency, and request-state pressure visible as separate constraints instead of letting one aggregate utilization chart dictate the fleet.