Embedding Model Infrastructure 2026: GPU Sizing and Cost for text-embedding-3, BGE-M3, and E5 at RAG Scale

At 100M daily embeddings, the API bill from OpenAI text-embedding-3 runs $2,000-$13,000 per day. That is the number that sends ML engineers to the self-hosting conversation, and it happens faster than most teams expect. By the time your RAG system is genuinely useful to users, you are indexing more content than anyone budgeted for - and the embedding cost is no longer a rounding error.

The open-source landscape has consolidated around a few clear winners. BGE-M3 from BAAI is the dominant self-hosted choice in 2026: 567M parameters, encoder-only, 1024-dimensional output, genuinely multilingual. At FP16, the model weights occupy ~1.1GB of VRAM, meaning you can fit six parallel instances on a single L4 with room for the serving framework. E5-large-v2 from Microsoft (335M params, ~670MB VRAM at FP16) is the lighter English-only alternative when query latency is the priority. GTE-large from Alibaba is similar in architecture and competitive on retrieval benchmarks - useful as a drop-in alternative if BGE-M3 underperforms on your specific data distribution.

The complication in 2026 is GTE-Qwen2-7B-instruct - a 7B decoder-only model fine-tuned for embeddings. It delivers materially better quality on long-document retrieval and complex semantic similarity tasks, but at 15GB VRAM and the throughput characteristics of a small generative LLM rather than an encoder. If your retrieval quality demands justify it, GPU requirements double or triple versus BGE-M3. Most teams running production RAG at scale do not need it.

Embedding model GPU requirements in 2026 defy the mental model most engineers carry from LLM deployments. Generative models are memory-bandwidth-bound: they generate one token at a time autoregressively, and the bottleneck is how fast you stream model weights from HBM into compute units. BGE-M3 and E5-large-v2 are encoder-only - they run a single forward pass over your entire input, produce one vector, and finish. The GPU profile is closer to image classification inference than to GPT decode.

This matters for GPU selection because compute-bound workloads favor raw TFLOPS over memory bandwidth - exactly where mid-tier GPUs like the L4 and A10G shine relative to their price. An H100 SXM5 has 3.35 TB/s HBM3 bandwidth, which is transformational for autoregressive decode. For BGE-M3 running batch-64 inference, most of that bandwidth advantage disappears because you are not streaming weights token by token. You pay for capability you cannot use with encoder models.

The practical consequence: embedding models scale horizontally much more cleanly than LLMs. Running 8-12 parallel BGE-M3 instances on a single L4 allows you to serve independent queries simultaneously with zero inter-GPU communication. No tensor parallelism, no NVLink requirements, no expensive HGX chassis. Two L4 nodes behind a load balancer often beats one A100 at this workload type, at 40% lower total cost. The embedding model GPU sizing calculation is almost entirely about aggregate throughput per dollar, not peak single-GPU performance.

The following numbers reflect BGE-M3 at FP16 with Text Embeddings Inference (TEI), sequence length 256 tokens. These represent a properly tuned production deployment - not the baseline you get from a quick Python script with sentence-transformers and no Flash Attention. The gap between a naive deployment and a tuned TEI instance is typically 3-5x throughput for encoder models.

The L4 to A10G throughput delta holds consistently around 30%, while A10G cards run 50-60% higher per hour. Unless your embedding workload genuinely requires that throughput margin, you are paying a premium the workload does not justify. The H100 reaches roughly 5x the L4 throughput - whether that premium pays off is a function of your daily volume and whether you need low-latency continuous availability or can burst-process in shorter windows.

At batch sizes above 256, throughput gains flatten on all three GPUs. Moving from batch 64 to batch 512 delivers roughly 40% more throughput on L4, but P99 latency per batch grows from ~50ms to ~350ms. The throughput gains are real, but the latency cost means you cannot run batch 512 on user-facing synchronous endpoints. Two deployment configurations - one tuned for async indexing throughput, one tuned for low-latency query embedding - extract more value from any GPU than a single instance trying to handle both.

The cost-per-million-embeddings calculation is where embedding infrastructure decisions get counterintuitive. Most engineers assume H100 - the fastest GPU - is also the best value per embedding. For LLM inference that is often true because FP8 compute and memory bandwidth both work in H100's favor. For BGE-M3 embedding inference, the calculation reverses.

At $0.42/hr sourced through ClusterBid, an L4 running BGE-M3 at batch 512 produces 7,400 sent/sec x 3,600 = 26.6M embeddings per hour. That works out to $0.016 per million embeddings. An H100 SXM5 at $2.50/hr produces 38,000 x 3,600 = 136.8M per hour - $0.018 per million embeddings. The L4 is cheaper per embedding than the H100. The H100 produces more per hour (useful when you need raw throughput headroom to absorb spikes), but on a per-unit cost basis, the L4 wins this workload.

At 100M daily embeddings - a realistic scale for a mid-size enterprise RAG system - two L4 nodes at $0.42/hr each running continuously cost $20/day. One H100 node handling the same volume in batch bursts costs $1.83/day in compute but $60/day if held on reserved capacity for low-latency serving. The right answer depends on your serving model: batch-only overnight indexing favors on-demand H100 bursts; continuous low-latency embedding service favors always-on L4 nodes. ClusterBid can source L40S and A10G capacity from smaller datacenters at rates the hyperscalers do not offer - the L4 in particular has been consistently available below $0.50/hr from inventory we source from regional DCs.

Batch size is the highest-leverage configuration knob in embedding infrastructure. Teams that set batch_size=1 and never revisit are leaving 30-50% GPU throughput unused. Teams that set batch_size=512 for a user-facing search endpoint are giving their users 400ms P99 on a query that should complete in 15ms. Both mistakes are common, and both are expensive - the first in wasted GPU time, the second in user experience and SLA violations.

The general rule for embedding model GPU sizing: for async document indexing pipelines, maximize batch size. Batch 256-512 is appropriate when documents arrive in bulk and end-to-end indexing time matters more than individual document latency. P99 latency per batch at batch=512 on an L4 is 300-400ms, but you are processing 512 documents - individual document latency is under 1ms. For synchronous user-facing embedding (the query side of your RAG system), batch size 1-8 is usually right. P99 at batch=1 for BGE-M3 on L4 is 8-15ms. At batch=8, P99 is 20-40ms - still fast enough for most applications, and you benefit from accumulated concurrent requests during peak traffic.

The production architecture that handles both workloads cleanly: two separate TEI instances with different batch configurations rather than one instance trying to serve both. Your document indexing pipeline hits the high-batch instance via an async queue. Your query API hits the low-batch instance directly. Two L4 nodes configured differently often outperform a single A10G trying to balance both patterns. TEI supports max_batch_tokens and max_concurrent_requests per instance, so you can differentiate without running separate containers on separate hardware - though separate nodes give you true workload isolation.

Three serving architectures are viable in 2026 for embedding model GPU deployment. The right choice depends on what you already operate, not which framework has the better benchmark headline this week. All three can hit the throughput numbers in section 03 when configured correctly. The differences are in operational overhead, feature coverage, and how they compose with the rest of your inference stack.

Text Embeddings Inference (TEI) from Hugging Face is the purpose-built option and the default recommendation for dedicated embedding serving. TEI is optimized specifically for encoder models: Flash Attention 2 backend, token-level batching, and a Candle kernel that avoids Python overhead in the hot path. BGE-M3 support including late interaction scoring for ColBERT-style retrieval was added in TEI v1.2. Deployment is a single Docker image with an HTTP/gRPC API. Prometheus metrics ship out of the box. NVIDIA GPU support includes L4, A10G, and H100 without any configuration changes. If you are building new embedding infrastructure from scratch, this is where to start.

vLLM added embedding model support in 0.4.x and has matured significantly. The main advantage is operational uniformity - if you are already running vLLM for your generative models, adding embedding endpoints to the same fleet eliminates a second deployment pipeline and a second observability stack. Throughput is within 10-15% of TEI for encoder models. The tradeoff: vLLM's PagedAttention memory management is designed for autoregressive decode, and you need to tune max_num_batched_tokens more carefully to get encoding-optimized throughput. For teams with mature vLLM infrastructure, this path has the lowest operational cost. Custom sentence-transformers deployments work but leave 3-5x throughput on the table versus a tuned TEI instance - if you inherited one of these, benchmark it against TEI before building more infrastructure around it.

Under 50M embeddings per day, a single L4 node handles the workload with more than 80% idle capacity remaining when running continuous low-latency serving. You pay for 24 hours of availability to keep P99 latency low - at $0.42/hr through ClusterBid, that is $10.08/day or $305/month including both query and indexing workloads. Add a second L4 for redundancy and you are at $610/month with zero single-point-of-failure risk. This is the right architecture for most early-stage and growth-stage RAG deployments.

At 50M-500M embeddings per day, horizontal L4 scaling is the right call. Two to eight L4 nodes behind a load balancer achieve the aggregate throughput at lower total cost than A10G or H100 clusters. A10G cards are 30% faster per card but 55% more expensive per hour - you would need to run significantly fewer nodes to break even, and for embedding workloads, fewer-but-faster rarely beats more-but-cheaper. The exception is space-constrained colocation where per-node costs (power, rack space) are high enough that fewer physical nodes save more than the GPU hourly premium costs.

At 500M+ embeddings per day, or when you need to run multiple embedding models simultaneously, H100 makes operational sense. Not because it is cheaper per embedding - the L4 still wins that comparison at most price points - but because at this volume you are managing 10-20 L4 nodes versus 3-4 H100 nodes. Fewer orchestration targets, fewer failure modes, and H100's 80GB VRAM lets you colocate multiple embedding models without GPU-to-GPU context switching. If you are already running H100 clusters for generative inference, adding embedding workloads to spare VRAM capacity on existing nodes is effectively free throughput. ClusterBid inventories L40S and A10G from smaller datacenters at rates the major hyperscalers do not match - if you want L4 capacity without queuing for cloud spot, check the inventory page for current sourcing rates.