Inference Quantization Showdown 2026: NF4 vs GPTQ vs AWQ

Quantization is the single most effective cost lever for production LLM inference. Dropping from FP16 to FP8 cuts memory and compute requirements in half. Moving to INT4 or NF4 can halve them again, but the trade-offs between accuracy, throughput, and hardware compatibility are nuanced.

Three methods dominate production deployments in 2026: NF4 (NVIDIA's native FP4 on Blackwell), GPTQ (the post-training workhorse), and AWQ (activation-aware quantization). Each has distinct strengths depending on model architecture, GPU generation, and latency targets.

NF4 is exclusive to NVIDIA's Blackwell architecture (B200 and B300). It leverages the Transformer Engine's native FP4 tensor core support, meaning quantized matrices run through dedicated hardware paths rather than software emulation. The result is up to 2x throughput over FP8 on transformer-based models with no additional kernel overhead.

Quality retention on NF4 is surprisingly strong. On MMLU-Pro and HumanEval, B200 running NF4 loses less than 0.8% accuracy versus FP16 for models up to 70B parameters. Past 100B, the gap widens to 1.5–2%, making NF4 viable for most production workloads except those requiring highest-possible fidelity.

GPTQ remains the most widely deployed quantization method because it works on every GPU generation from A100 onward. It performs one-shot weight quantization using an approximate second-order optimization, minimizing the MSE of layer outputs on a small calibration dataset. A 70B model can be quantized to 4-bit in roughly 4 hours on a single H100.

The trade-off is throughput. GPTQ INT4 requires custom CUDA kernels (typically via GPTQ-for-LLaMA or AutoGPTQ) that achieve 60–70% of theoretical peak on H100. On B200, the gap is larger - roughly 45–55% of peak - because the kernels cannot exploit Blackwell's FP4 tensor cores. GPTQ's strength is compatibility, not raw speed.

AWQ improves on GPTQ by analyzing activation patterns to identify which 1% of weights are disproportionately important. Those salient weights are kept at higher precision (FP16) while the rest are quantized to INT4. This selective retention reduces the perplexity gap versus FP16 by roughly 40% compared to GPTQ at the same average bit width.

Throughput on AWQ is roughly on par with GPTQ on H100 (within 5–10%) and slightly better on B200 thanks to better kernel fusion in the vLLM and SGLang runtimes. The main cost is the additional calibration pass, which takes 6–8 hours for a 70B model versus GPTQ's 4 hours.

The decision matrix for choosing a quantization method depends on which GPU you are running, your accuracy requirements, and your throughput targets. The table below summarizes results on Llama-3.1 70B with batch size 32 on single-GPU configurations.

All results measured with the vLLM 0.8 runtime using the same prompt dataset. Perplexity delta measured against FP16 baseline on WikiText-2.

Hardware compatibility is the gating factor. NF4 is Blackwell-only - period. If you are on H100 or A100 clusters, NF4 is not available regardless of software stack. GPTQ and AWQ run on any NVIDIA GPU with compute capability 7.0+ (V100 and later), though throughput scales with tensor core generation.

On AMD MI300X, GPTQ and AWQ run via ROCm and the vLLM AMD backend, but throughput is typically 15–25% lower than NVIDIA equivalent due to less optimized ROCm kernel libraries. NF4 has no AMD equivalent as of mid-2026.

For teams on Blackwell infrastructure serving latency-sensitive production inference, NF4 is the default choice. The 2x throughput gain over INT4 methods directly translates to lower cost per token, and the accuracy penalty is negligible for chat, code generation, and summarization.

For teams on Hopper or Ampere clusters, or those running models above 100B parameters where NF4 accuracy degrades, AWQ offers the best accuracy-efficiency balance. GPTQ remains the pragmatic choice when calibration time matters or when deploying across heterogeneous GPU fleets.