The AI Memory Arena (AIMA) Benchmark: How GPU Memory Bandwidth and Capacity Drive Inference Accuracy at Scale

The AI Memory Arena (AIMA) benchmark, published by MLCommons in early 2026, is the first standardized benchmark designed specifically to isolate GPU memory subsystem performance in inference workloads. Unlike MLPerf, which measures end-to-end throughput and latency, AIMA measures how memory bandwidth and capacity affect the accuracy and consistency of inference results under memory pressure.

AIMA works by running inference on progressively larger batch sizes and sequence lengths until the GPU runs out of memory, then measuring the accuracy degradation that occurs as the memory subsystem approaches saturation. The benchmark produces three scores: the memory bandwidth efficiency score (how well the GPU utilizes its peak HBM bandwidth under real inference conditions), the memory capacity ceiling (the maximum batch-size-sequence-length product the GPU can handle before OOM), and the accuracy consistency index (the variance in output quality across memory pressure levels).

The AIMA bandwidth efficiency score reveals that peak theoretical memory bandwidth is rarely achieved in real inference workloads. The H100 SXM achieves 3.35 TB/s peak HBM3 bandwidth but sustains an average of 2.1 TB/s (63% efficiency) on the AIMA attention-intensive workload. The H200 with HBM3e reaches 4.8 TB/s peak but sustains 3.4 TB/s (71% efficiency). The B200 achieves 8.0 TB/s peak with 6.2 TB/s sustained (78% efficiency). The B300 leads at 12 TB/s peak and 10.1 TB/s sustained (84% efficiency).

The efficiency improvement across generations is not just about faster HBM. Blackwell's improved memory controller and the larger L2 cache (80 MB vs 50 MB on Hopper) reduce the number of HBM accesses needed. The L2 cache hit rate on the AIMA workload is 42% for B300 vs 31% for H200. Each percentage point of L2 hit rate improvement translates to roughly 40 GB/s of effective bandwidth gain.

The AIMA memory capacity ceiling score measures the maximum workload a GPU can fit before running out of memory. For Llama 3.1 70B at FP16 with 32k context, the H100 reaches its ceiling at batch size 8 (using 78 GB of 80 GB, with KV cache consuming 23 GB). The H200 reaches batch size 14 (using 132 GB of 141 GB). The B200 reaches batch size 24 (using 170 GB of 192 GB). The B300 reaches batch size 38 (using 260 GB of 288 GB).

The practical implication: the B300 can serve 4.75x the concurrent users of an H100 for the same model and context length, without trading off accuracy through quantization or context reduction. For production deployments at scale, this capacity advantage translates directly into lower cost-per-token and simpler serving infrastructure with fewer nodes.

The AIMA accuracy consistency index measures how much output quality degrades as the memory subsystem approaches saturation. This is the benchmark's most novel contribution: as GPU memory utilization exceeds 85-90%, the memory controller's queuing delays cause some attention computations to use stale or incomplete KV cache data, producing measurable accuracy degradation even without explicit quantization.

The H200 shows an accuracy consistency index of 0.92 (1.0 is perfect consistency), meaning output quality degrades by up to 8% at high memory pressure. The B200 scores 0.95, and the B300 scores 0.97. The improvement comes from Blackwell's larger L2 cache, which buffers frequently accessed KV cache entries and reduces the probability of memory controller contention. For latency-sensitive applications like real-time chat or code generation, a 5-8% accuracy variance under load is noticeable in production quality monitoring.

Memory contention directly affects P99 inference latency. When the memory subsystem is saturated, queuing delays at the HBM controller add 15-40% to attention computation time. The AIMA benchmark measures P99 latency at three memory pressure levels: low (50% utilization), medium (75%), and high (90%).

At high memory pressure, the H100 shows a 2.1x increase in P99 latency versus low pressure. The H200 shows 1.7x, the B200 shows 1.4x, and the B300 shows 1.25x. The B300's 1.25x ratio means that P99 latency stays within 25% of the no-pressure baseline even as the GPU approaches capacity. This consistency is critical for production SLAs: a serving infrastructure built on B300 GPUs can operate at higher utilization levels without violating P99 latency targets, which directly improves GPU utilization and reduces cost.

AIMA scores change the GPU buying calculus in three ways. First, memory bandwidth efficiency is a better predictor of inference performance than raw peak bandwidth. The B300's 84% efficiency versus H100's 63% means the real-world performance gap is wider than the peak bandwidth numbers suggest. Second, the accuracy consistency index explicitly measures a dimension that MLPerf ignores: how reliable is the quality of output under load. For customer-facing inference, this is arguably more important than peak throughput.

Third, the capacity ceiling score reveals the real cost of memory-limited deployments. A B300 serving batch size 38 replaces roughly 4.75 H100 GPUs (at batch size 8 each) for the same workload. At current spot pricing, the B300 costs approximately $5.50/GPU/hr versus $2.20/GPU/hr for H100. The cost-per-batch-slot comparison: $0.145/batch-slot/hr for B300 vs $0.275/batch-slot/hr for H100 -- a 47% cost advantage for B300 on memory-bound inference.

Different workloads benefit from different AIMA dimensions. Long-context LLM inference (128k+ tokens) is dominated by the memory capacity ceiling score: the GPU that can fit the largest KV cache wins. Mixture-of-experts models benefit most from bandwidth efficiency, since their sparse activation pattern creates irregular memory access patterns that stress the memory controller. Dense small models (7B-13B) at high batch sizes benefit from accuracy consistency, since the primary bottleneck is memory controller contention at high utilization.

The AIMA benchmark provides workload-specific subscores: for MoE models, the B300 scores 91% bandwidth efficiency versus H200's 68%. For dense models, the gap narrows: B300 at 86% versus H200 at 74%. For long-context workloads, the B300's capacity ceiling is 4.2x H200's. These subscores enable procurement teams to match GPU hardware to workload profiles rather than buying a single GPU type for all workloads, which is the standard approach today.

AIMA has three limitations to consider. It measures memory subsystem performance in isolation, so GPU compute-bound workloads (low batch size, high precision training) are not well represented by AIMA scores. It uses a single model architecture (a 70B dense transformer) as the reference workload, so MoE and vision model performance must be inferred from the subscores rather than directly measured. And it does not account for software optimization: a well-tuned vLLM deployment on H200 can outperform an untuned B300 deployment despite lower AIMA scores.

The best procurement strategy combines AIMA scores with MLPerf v5.1 results and workload-specific benchmarking. Use AIMA for memory-bound inference sizing, MLPerf for peak throughput comparison, and your own models for final validation. The AIMA benchmark is a new addition to the evaluation toolkit, not a replacement for existing benchmarks. We recommend running it as a pre-screen before committing to a multi-month GPU reservation.