What Does 8x Actually Mean?
The 8x figure floating around FP4 compares peak dense FP4 throughput against a lower-precision-unfriendly baseline, not your production cost per token. Your real savings come from memory footprint plus kernel efficiency at your batch size. For most serving setups that lands between 1.5x and 3x on $/M tokens.
Anyone quoting peak FLOPS as a cost cut without batch size, context length, and accuracy gates is selling the benchmark, not the outcome. Ask for the serving config or ignore the number.
Where Does FP4 Hurt Accuracy?
FP4 is kindest to large dense models with smooth activation distributions and hardest on small models, MoE routers, and long-context retrieval where quantization noise compounds. In our experience: 70B+ chat and summarization survive FP4 well; sub-14B agents with tool calling need per-layer validation.
Gate every migration on your own evals: MMLU-style knowledge, plus a task-specific set (function-call accuracy, citation faithfulness). If FP4 drops tool-call reliability by even 2 points, the GPU savings evaporate into retries.
| Workload | FP4 verdict | Gate metric |
|---|---|---|
| 70B+ chat serving | Usually safe | Human preference delta |
| RAG + citations | Validate per layer | Faithfulness score |
| Tool-calling agents | Often risky | Call accuracy |
Which Serving Stacks Support NVFP4 Today?
TensorRT-LLM and recent vLLM builds carry the working NVFP4 paths, with model-by-model kernel coverage. Coverage is the catch: your exact model and attention variant must be in the supported matrix, or you fall back to slower mixed paths.
Before committing, run the supported-model check plus a 24-hour soak at production traffic shape. FP4 bugs show up as tail-latency spikes and rare NaN cascades, not as clean failures.
The Honest $/M Token Math: FP8 vs FP4
On B200, moving a 70B model from FP8 to FP4 typically cuts KV and weight memory enough to raise max batch 1.5x to 2x at fixed latency SLO. That batch expansion, not the peak-FLOPS ratio, is what lowers $/M tokens. Pricing referenced here reflects market conditions at time of writing and can fluctuate with GPU availability.
Our rule: model the cut as 30 to 50 percent on memory-bound workloads, 10 to 20 percent on compute-bound short-context workloads. Then validate with a week of shadow traffic before touching the production pool.
FP4 Migration Checklist That Prevents Rollbacks
Freeze the eval set first: accuracy gates, latency SLOs (p50/p95/p99), and throughput floors. Quantize per-module, keep the router and embeddings at higher precision on MoE builds, and canary on 5 percent of traffic with automatic rollback on gate breach.
Log per-token acceptance and per-layer scales. When FP4 regresses, the fix is almost always selective: lift two or three sensitive layers back to FP8 and keep the rest at FP4. Full rollback is the expensive admission that you skipped the canary.
When Should You Just Stay on FP8?
Stay on FP8 when your bottleneck is prefill compute at short context, when kernel coverage for your model is incomplete, or when your evals show any reliability regression you cannot bound. FP8 on B200 is already excellent; FP4 is an optimization, not a rescue.
If you want to price the two paths on current rental supply before re-quantizing your fleet, compare B200 inference listings on ClusterBid and run both precisions against the same $/M token target.