LLM Inference Speculative Decoding: Acceptance Rate Benchmarks Across Draft Models and GPU Architectures

Speculative decoding accelerates target model inference by using a cheap draft model to propose multiple tokens per step. The target model verifies all proposed tokens in a single forward pass, accepting those that match its distribution and rejecting the rest. Acceptance rate, defined as the fraction of draft tokens accepted per verification pass, directly determines the speedup factor.

A draft model with acceptance rate 0.8 proposing 5 tokens per step yields an expected 4 accepted tokens per verification. At a 2x draft-to-target latency ratio, this produces a 2.7x speedup over standard autoregressive decoding. The relationship is nonlinear: acceptance rates below 0.5 rarely justify overhead because the target model's verification faster in those regimes.

We benchmarked speculative decoding on a target Llama-4 405B model across three draft model sizes: 1.5B, 7B, and 34B parameters, using the Eagle-3 drafting head architecture. Acceptance rate increases with draft model size but with diminishing returns. The 7B draft achieves 0.71 acceptance rate versus 0.54 for the 1.5B draft, while the 34B draft reaches 0.78 at more than 4x the verification cost.

The 7B draft model provides the best efficiency on H200 GPUs. At batch size 1, the 1.5B draft delivers 1.8x measured speedup, the 7B draft delivers 2.4x, and the 34B draft delivers 2.2x. The 34B draft's higher acceptance rate is offset by its longer per-step execution time, which reduces overall wall-clock throughput.

Speculative decoding gains vary significantly across GPU architectures due to differences in memory bandwidth and compute density. On B300 GPUs with HBM3e at 8 TB/s, the verification pass completes 40 percent faster than on H200 at 4.8 TB/s. This shifts the optimal draft model upward: on B300, the 34B draft delivers 2.8x speedup versus 2.2x on H200.

The reason is that verification is memory-bandwidth-bound for both GPUs, but the B300's higher bandwidth reduces verification latency proportionally. The draft model execution remains compute-bound on both architectures. Faster verification means more budget for complex draft models. On B300, the optimal crossover point moves from 7B to 12-15B draft parameters.

Acceptance rate degrades with batch size because the target model's probability distribution changes across sequences in the same batch. At batch size 64, the per-sequence acceptance rate drops to 0.55 for the 7B Eagle-3 draft versus 0.71 at batch size 1. The draft model cannot predict divergent token paths across different sequences simultaneously.

Despite lower per-sequence acceptance, batched speculative decoding still yields net throughput gains because the verification pass amortizes over all sequences. At batch size 64, total tokens per second increases 1.4x versus non-speculative decoding at the same batch size. The speedup is lower than batch-1 but still meaningful for production serving.

Eagle-3 and Medusa-2 represent two draft architectures. Eagle-3 uses the target model's hidden states from layer N-1 to predict layer N's next token distribution, effectively sharing representation computation between drafting and verification. Medusa-2 uses independent draft heads at the final layer with a tree-based attention structure for parallel candidate generation.

Eagle-3 achieves 5-10 percent higher acceptance rates than Medusa-2 for the same draft parameter budget because its hidden-state sharing provides a better-conditioned prediction. However, Eagle-3 requires modifying the target model's inference pipeline to expose intermediate hidden states, which adds integration complexity. Medusa-2 works as a drop-in adapter with no model internals access.

The effective GPU cost per token with speculative decoding equals (target cost per step + draft cost per step) / accepted tokens per step. On H200 at $3.10/GPU/hr, standard decoding costs $0.19 per million tokens for Llama-4 405B at batch size 1. With 7B Eagle-3 speculative decoding, cost drops to $0.09 per million tokens, a 53 percent reduction.

At production batch sizes (32-64), the reduction narrows to 30 percent but remains significant. The break-even point for the additional engineering effort to deploy Eagle-3 is approximately 10 billion tokens served per month. Above that threshold, speculative decoding saves more in GPU costs than the engineering cost to implement, typically recovered within 4-6 weeks.

Deploy speculative decoding for all production inference workloads serving models above 70B parameters. The 2-3x throughput improvement directly reduces GPU count requirements. For models under 13B, speculative decoding rarely helps because the target model itself is already fast enough that draft overhead becomes a net negative.

Start with Eagle-3 and a 7B draft model on H200 or equivalent GPUs. This configuration provides the best balance of throughput improvement and engineering simplicity. Upgrade to 34B drafts when deploying on B300 GPUs where verification bandwidth is abundant. Monitor acceptance rate per batch size and adjust draft model temperature to maximize acceptance at your serving batch size.