Remove waste before adding cleverness
The ladder moves from low-risk capacity hygiene to workload-specific engine and hardware changes. Stop when the target is met.
- Define workload and SLO.Without the useful unit, optimization has no objective function.
- Capture a reproducible baseline.Trace, configuration, quality, tail latency, goodput, memory, energy, and fully loaded cost.
- Delete idle allocation.Expire abandoned notebooks/endpoints, stop orphaned nodes, consolidate tiny pools, and make ownership visible.
- Right-size device and count.Choose the smallest complete configuration that passes memory, topology, SLO, and availability.
- Improve scheduling and reuse.Continuous batching, token budget, chunked prefill, stable templates, prefix/session routing.
- Reduce bytes.Weight/activation/KV precision, fusion, graphs, checkpointing, offload where transfers are favorable.
- Change topology.Reduce unnecessary TP, place collectives correctly, add DP, or disaggregate only after evidence.
- Share and autoscale.Partition predictable small work; scale on queue/token/SLO signals with warm-capacity policy.
- Optimize purchase model.Commit only the proven floor; use flexible and interruptible supply for the right tail.
- Tune power last.Find throughput-per-watt or joules per useful unit while preserving the SLO.
Say what is quantized
Weights, activations, KV cache, and native low-precision formats have different benefits and quality risks.
| Component | Example | Primary benefit | Primary risk |
|---|---|---|---|
| Weights only | W4A16, W8A16, AWQ, GPTQ | Model fit and lower decode weight traffic | Dequant/kernel overhead; quality; little prefill gain |
| Weights + activations | W8A8, FP8 | Less traffic and accelerated GEMMs on native hardware | Calibration and format/hardware dependence |
| KV cache | FP8/INT8/INT4 KV | More context/concurrency and lower cache traffic | Recall/quality loss and conversion overhead |
| Native very-low precision | NVFP4, MXFP4 | Large recent-hardware capacity/compute opportunity | Architecture-, model-, scale-, and kernel-specific |
- Establish BF16/FP16 quality and performance.This is the reference, not an assumption.
- Try native FP8.Only where GPU, model, and engine expose optimized kernels.
- Try weight-only 4-bit.When residency or low-batch decode bandwidth is the constraint.
- Quantize KV separately.When long context or concurrency - not weights - is the limit.
- Run task quality gates.Accuracy, structured output, long-context recall, safety, and latency on the real distribution.
Caching and speculation win only when reuse or acceptance is real
Prefix cache
Reuse token-identical prompt blocks. Stabilize system prompts and tool schemas, preserve tokenizer/template versions, route related sessions together, and track hit tokens - not only hit requests.
KV offload
Move reusable or cold blocks to host memory only when saved recomputation exceeds transfer and eviction cost. PCIe, coherent links, and CPU memory bandwidth produce different results.
Speculative decode
Let a draft, n-gram, suffix, MTP, or other proposer guess tokens; the target verifies them. It tends to help low/medium-batch decode with high acceptance and unused compute.
Template engineering
Reordered JSON keys, changing timestamps, whitespace, and per-request tool definitions can eliminate prefix reuse. Treat token stability as a measurable performance feature.
Every optimization needs a falsifiable scorecard
| Dimension | Baseline and candidate must record |
|---|---|
| Identity | Model/app commit, dataset/trace version, tokenizer, engine/container, driver and firmware |
| Hardware | Exact GPU variant/count, host, topology, NIC, power cap, cooling, placement |
| Load | Input/output distributions, arrival rate/burstiness, concurrency, warm/cold state, random seed |
| Service | p50/p95/p99 latency, throughput, errors, goodput and cold-start |
| Quality | Task accuracy, output validity, long-context and safety gates appropriate to the application |
| Resources | Peak HBM, KV occupancy, queue, tensor/SM, DRAM, link, CPU, power and energy |
| Economics | Fully loaded $/useful unit plus engineering and operational complexity |
| Decision | Predeclared pass threshold, confidence/repeats, keep/revert owner and date |