vLLM Multi-LoRA Serving Economics: Hundreds of Adapters on One GPU

Fine-tuning large language models via LoRA has become the default adaptation strategy for most production deployments. The technique works: train a small set of rank-decomposition matrices (typically rank 8-128) while freezing the base model weights, yielding adapter weights that are 1-2% of the full model size. A Qwen 2.5 72B model compressed to rank-64 LoRA weighs roughly 640MB per adapter - trivially swappable. The operational problem emerges when you have many of them.

Consider a typical AI SaaS company: they serve 150 customer tenants, each with a custom LoRA adapter fine-tuned on their domain data. The naive deployment is 150 dedicated model replicas on H100 clusters. At $1.15/hr per H100 SXM5 on ClusterBid's marketplace, each replica running on a single GPU (for sub-70B models) costs $1.15/hr. Running 150 replicas: $172.50/hr, or $124,200/month. Most Series B teams cannot sustain that burn. The headroom between total GPU memory and what the base model actually needs is the wasted resource.

The base model - whether Llama 3.1 70B or Qwen 2.5 72B - occupies roughly 35-40GB at FP16 (or 20-25GB at FP8). An H100 SXM5 has 80GB HBM. That leaves 40-60GB per GPU sitting idle. Multi-LoRA serving architectures exploit exactly this gap: instead of replicating the full model per tenant, they load the base model once and dynamically swap adapter weights in the same GPU memory, serving hundreds of tenants from fewer GPUs.

vLLM's PagedAttention solved the memory fragmentation problem for KV caches by managing attention key-value blocks in a paging system analogous to virtual memory. Each request's KV cache is split into fixed-size blocks that can be stored non-contiguously in GPU memory, dramatically reducing fragmentation and enabling higher batch utilization. This is the foundation that makes multi-adapter serving feasible - without it, KV cache fragmentation would make the memory management of swapping adapters too expensive to be practical.

S-LoRA extends vLLM with a co-designed adapter management system. The core insight: adapter weights for active requests are loaded into GPU memory on-demand, while inactive adapter weights stay in CPU pinned memory. The S-LoRA scheduler coordinates which adapters are preloaded and which are evicted, prioritizing adapters expected to serve upcoming requests. The overhead of swapping an adapter between CPU and GPU is roughly 15-25ms for a rank-64 adapter on a PCIe 5.0 link - small enough to be invisible at typical serving latencies of 200-500ms per request.

The system maintains a unified memory pool shared between the base model weights, the active LoRA adapters, and the KV cache blocks. vLLM's memory manager tracks all three categories and can dynamically reallocate blocks as workloads shift. When a new tenant request arrives for an adapter not in GPU memory, S-LoRA's prefetcher initiates the transfer while the base model computation proceeds, effectively hiding the swap latency behind compute.

The architecture is not magic - it requires the base model to fit in GPU memory with enough headroom for at least 2-4 adapters and KV cache blocks. For a 70B model at FP8 on an H100 SXM5 (80GB), the model takes ~20GB, leaving ~60GB. With a typical adapter at 640MB and KV cache requiring ~4GB per active request (4k context), a single GPU can comfortably serve 4-6 concurrent requests with 8-12 adapters preloaded. For the full breakdown of how H100 memory budgets work across serving configurations, see our H100 memory and serving guide.

The cost arithmetic is straightforward. A dedicated replica approach for 150 LoRA adapters on a 70B base model requires at minimum 19x H100 SXM5 GPUs to cover 150 tenants (assuming 8 tenants per GPU using model parallelism). At $1.15/hr per GPU (ClusterBid on-demand mid-2026), that is $21.85/hr or roughly $15,732/month.

The S-LoRA / multi-LoRA approach as implemented in vLLM 0.8+ can handle the same 150 adapters on 4x H100 SXM5 GPUs, serving approximately 20-24 concurrent requests at reasonable latency. At $4.60/hr total, that is $3,312/month - roughly 4.8x cheaper than the dedicated replica approach. The savings compound as the number of adapters grows because the base model cost is amortized across all tenants.

The breakeven point is around 15-20 adapters. Below 15 adapters, the overhead of adapter management and the complexity of multi-LoRA scheduling may not justify itself versus simple replica allocation. Above 50 adapters, multi-LoRA is definitively cheaper. Above 200 adapters, it is the only economically viable approach.

The limiting resource in multi-LoRA serving is not compute - it is the memory budget for simultaneously loaded adapters. Each adapter consumes memory proportional to its rank and the size of the linear layers it modifies. For Llama 3.1 70B with rank-64 LoRA applied to all attention and FFN layers, one adapter weighs approximately 640MB at FP16. An 80GB H100 SXM5 running the base model at FP8 (~20GB) with KV cache blocks (~4GB per active request) has roughly 56GB free for adapters and overhead. That supports approximately 80 adapters resident in GPU memory simultaneously if no requests are active, or about 12-16 adapters with typical concurrency of 4-8 concurrent requests.

Beyond the resident set, CPU pinned memory serves as the backing store. With 256GB of host memory (standard for an 8x H100 node), you can store approximately 400 adapter weights at 640MB each. The CPU-GPU transfer bandwidth over PCIe 5.0 x16 is roughly 32 GB/s theoretical, ~24 GB/s sustained. Swapping a 640MB adapter takes approximately 25ms. With S-LoRA's prefetch scheduler predicting which adapters the next batch of requests will need, the swap latency is typically hidden or amortized across multiple compute iterations.

The practical limit for the number of distinct adapters served from a single node is primarily a function of concurrency, not adapter count. If you have 1,000 tenants but only 20 concurrent requests at any moment, the system needs only 20 adapter slots on GPU (plus the prefetch buffer). The remaining 980 adapters live in host memory and get swapped in on demand. The key tuning knob is the adapter retention policy: how long an unused adapter stays in GPU memory before eviction. For most production workloads, a 30-60 second retention with LRU eviction provides the best hit rate without wasting GPU memory.

vLLM's multi-LoRA scheduler exposes several critical knobs. The first is max_num_batched_tokens, which controls how many tokens are batched per scheduling iteration. In a multi-LoRA setting, larger batches increase the likelihood of different adapters appearing in the same batch, which triggers more adapter swaps. A smaller batch size (512-1024 tokens) reduces swap pressure at the cost of lower GPU utilization. The optimal setting depends on your adapter count-to-concurrency ratio - more adapters per concurrency favors smaller batches.

The second critical parameter is the adapter load threshold. S-LoRA uses a scoring function based on request arrival rate, adapter hotness (recent request count), and tie-breaking by adapter size. Tuning the scoring weights to match your traffic patterns can reduce swap overhead by 30-50%. A tenant with predictable daily usage patterns - enterprise API consumers who hit the same endpoints at known hours - benefits from time-based prefetching that loads their adapter before the expected traffic spike.

The third knob is quantization of adapter weights. FP16 adapters at 640MB can be quantized to INT8 (320MB) or FP4 (160MB) with negligible accuracy impact for most fine-tuning use cases. At FP4, an H100 SXM5 can hold 300+ adapters in GPU memory simultaneously - effectively eliminating swapping for most deployments. The caveat: FP4 matrix multiplication for LoRA weights is not supported in all GPU architectures. H100's FP4 Tensor Core support (introduced via Hopper's FP4 path in CUDA 12.3+) makes this viable. B200 has native FP4 support. Older hardware may fall back to software dequantization, defeating the purpose.

Once you exceed the capacity of a single multi-LoRA node, you need a routing layer that directs requests to the node currently hosting the target adapter, or that can quickly load it. Two patterns dominate: hash-based routing and rendezvous hashing. Hash-based routing (adapter_id % N) is simple but causes cascading evictions when a node fails or a new node joins. Rendezvous hashing (consistent hashing) minimizes the number of adapters that need to relocate during cluster topology changes.

The routing layer must also handle adapter replication - popular adapters can be preloaded on multiple nodes to absorb traffic spikes. The replication factor is tunable per adapter, typically 2-3x for the top 10% of adapters by request volume. For very hot adapters serving 50+ RPS, routing every request through a single node creates a bottleneck at the adapter load path. Preloading those adapters on 3-4 nodes and routing via random selection spreads the load evenly.

Health checking in a multi-LoRA cluster is more nuanced than standard inference serving. A node may be healthy for the base model but have its adapter cache thrashed by a bad routing pattern. The routing layer should expose per-adapter latency metrics and dynamically adjust routing weights when a particular (node, adapter) pair shows elevated P99 latency. This feedback loop prevents cascading failures where a single adapter's traffic overload causes evictions that impact other adapters on the same node.

Multi-LoRA serving via vLLM 0.8+ is the right architecture when you have 20+ tenants with distinct fine-tuned adapters on a shared base model, concurrent request volume is below 200 RPS, and your latency budget allows occasional cold-start swaps of 25-50ms. This covers most B2B AI SaaS deployments: enterprise chatbots, document analysis tools, code generation assistants with per-organization fine-tuning, and vertical AI products serving regulated industries that require lightweight customization per customer.

Multi-LoRA is wrong when latency consistency is paramount and you cannot tolerate any swap overhead. Real-time voice assistants, low-latency trading agents, and interactive gaming AI typically require sub-50ms P99 for every request. In these cases, dedicated replicas for the top 5-10 hot adapters with S-LoRA handling the long tail is a practical hybrid: preload hot adapters on permanent GPU slots while the cold adapters time-share the remaining memory. This hybrid configuration is straightforward to implement by assigning adapter priority tiers in vLLM's scheduler configuration.

The sweet spot on ClusterBid's marketplace is the 4x H100 SXM5 configuration for 50-200 adapters: $4.60/hr on-demand, roughly 20-24 concurrent requests, and all adapters fitting in host memory. As your adapter count grows past 500, you scale horizontally by adding more 4x H100 nodes behind a consistent-hashing load balancer. The economic case compared to dedicated replicas only strengthens at scale - the price per additional tenant approaches zero once the base model infrastructure is paid for.