GPU Inference Benchmark Suite: Standardized Metrics for Model Serving Performance

Standardized inference benchmarking requires consistent measurement of four core metrics: time-to-first-token (TTFT), inter-token latency (ITL), end-to-end latency at P50/P95/P99, and throughput in tokens per second. For LLM serving, TTFT targets must stay under 500ms for interactive applications, with ITL below 30ms per token. A properly configured H100 serving Llama 3 70B with FP8 achieves approximately 1,200 tokens/second throughput at batch size 32, compared to 680 tokens/second on an A100 with FP16.

The MLPerf Inference v4.0 suite provides the closest industry standard, with benchmarks covering image classification, object detection, natural language processing, and recommendation systems. However, production workloads differ significantly from synthetic benchmarks. Real-world deployments see 30-50 percent lower throughput than MLPerf results due to networking overhead, request heterogeneity, and tail latency constraints. A benchmark suite must include both synthetic workloads and production trace replay to provide actionable data.

Production inference benchmarks must measure P50, P95, and P99 latency under sustained load. A system may average 50ms P50 latency but spike to 800ms at P99 under concurrent requests, which violates interactive SLAs. The ratio of P99 to P50 latency, known as the tail latency factor, should stay below 4x for well-tuned systems. NVIDIA TensorRT-LLM achieves tail latency factors of 2.5-3.5x on H100 with continuous batching, versus 4-6x on A100 without optimization.

Cost-per-token is the ultimate economic metric. At $3.78/hour H100 on-demand pricing, serving Llama 3 70B at 1,200 tok/s yields $0.00000315 per token. With FP8 quantization and optimized batching this drops to approximately $0.00000210 per token. In contrast, A100 at $2.50/hour with 680 tok/s yields $0.00000368 per token. The H100 achieves 43 percent lower cost-per-token despite 51 percent higher hourly cost.

Different inference workloads require tailored benchmark configurations. Chat applications with 2,048-token input and 512-token output stress prefill throughput differently than summarization workloads with 8,192-token inputs and 256-token outputs. Code generation workloads see high output-to-input ratios of 1:2, requiring sustained decode performance. A benchmark suite must parametrize input/output token ratios, batch sizes from 1 to 128, and concurrency levels from 1 to 256.

Benchmarking should also measure memory bandwidth utilization using NVIDIA bandwidthTest utility and custom kernels. H100 achieves 3.35 TB/s of HBM3 memory bandwidth, while A100 reaches 2.0 TB/s with HBM2e. Real-world inference typically achieves 60-75 percent of peak theoretical bandwidth. A system operating below 50 percent bandwidth utilization likely has a CPU-bound preprocessing pipeline, inefficient attention implementation, or PCIe bottleneck.

Reproducibility requires pinning software versions: CUDA 12.4, TensorRT-LLM 0.9.0, PyTorch 2.3.0, and specific kernel configurations. A single CUDA driver version change can alter inference latency by 5-15 percent. The benchmark pipeline must record the full software stack hash, GPU clock frequencies, power limits, and ambient temperature at test time. Temperature fluctuations of 5 degrees Celsius can cause GPU throttling that reduces throughput by 8-12 percent.

ClusterBid standardized benchmark harness runs on bare-metal instances across providers, reporting the full metric suite with confidence intervals. Each benchmark run executes 1,000 warmup iterations followed by 10,000 measurement iterations, with 95 percent confidence intervals calculated for all metrics. Results are published with the complete configuration to enable apples-to-apples comparison across providers.