A GPU is a throughput machine fed by a memory system
GPUs win when many operations can run in parallel. They lose when work is serial, kernels are too small, data cannot arrive fast enough, or communication dominates.
Matrix engines
Specialized units accelerate dense matrix operations at supported precisions and shapes. Peak tensor rates assume the right kernels, data type, and often a stated sparsity mode.
General cores and fixed-function blocks
Vector/scalar work, reductions, sampling, data transforms, codecs, and ray tracing can use different parts of the chip. Two workloads on one GPU may stress unrelated resources.
The memory hierarchy decides whether the job fits - and how often data moves
Capacity and bandwidth answer different questions: can the state reside, and can it be reused fast enough?
| Layer | What lives there | What to ask |
|---|---|---|
| Registers / on-chip SRAM | Current tiles, accumulators, frequently reused values | Does the kernel tile and fuse well, or spill? |
| HBM / GDDR | Weights, activations, optimizer state, KV cache, workspace | Does peak residency fit with safe headroom? Is the workload bandwidth-bound? |
| Peer fabric | Tensors, gradients, KV or activations crossing GPUs | How many bytes cross per layer or step, and on which topology? |
| Host RAM | Staging, offload, page cache, dataset and model loading | Is transfer time hidden, or is PCIe the bottleneck? |
| Network / storage | Checkpoints, remote weights, distributed collectives, inputs | What is cold-start, checkpoint, and recovery time? |
GPU memory = weights + KV or optimizer state + activations + workspace + graphs + runtime overheadPrecision changes fit, bandwidth, compute, and sometimes the answer
Lower precision is useful only when the hardware has efficient kernels and the workload still passes its quality gate.
| Format lane | Typical use | Benefit | Check |
|---|---|---|---|
| FP64 / FP32 | Scientific/HPC, numerically sensitive stages | Range and accuracy | Real solver tolerance; do not pay for FP64 if the job does not use it |
| TF32 / BF16 / FP16 | Training and high-quality inference baselines | Strong accelerator support; common reference point | Loss scaling, overflow, convergence, kernel support |
| FP8 / W8A8 | Modern training and inference | Less traffic and faster matrix math on supported GPUs | Vendor format, calibration, unsupported layers |
| W4A16 / INT4 | Weight-residency and decode-bandwidth pressure | Larger model or batch in the same HBM | Dequant overhead, task quality, shape-specific kernels |
| FP4 / microscaling | Recent hardware with native low-precision paths | Large capacity and throughput opportunity | Format is architecture-specific; validate model, scale metadata, and engine |
Dense and sparse rates are not interchangeable. NVIDIA product pages often headline structured-sparse peak rates and state dense performance is half; AMD typically separates dense and structured-sparse rows. Never divide one vendor’s sparse FP8 number by another vendor’s dense FP8 number.
File size is not runtime behavior. Quantization can add scale metadata, leave some layers at higher precision, or fall back to a slower kernel. Measure memory, quality, and end-to-end speed together.
Ask what the next unit of performance is waiting for
Arithmetic intensity - the useful operations performed per byte moved - is the shortest route to the right optimization.
Compute-bound
Matrix/tensor activity is high and larger/faster compute or lower-precision arithmetic helps. Common in long-prompt prefill and large training GEMMs.
Memory-bound
HBM activity is high while compute is underfilled. Batching, weight/KV quantization, fusion, and cache reuse may help. Low-batch decode often lives here.
Communication-bound
Peer or network collectives dominate. Reduce shard count, improve placement, overlap transfers, or buy a topology - not merely a faster GPU.
- Profile a real input shape and concurrency, not a synthetic matrix alone.
- Check host preprocessing, tokenization, storage, and network before blaming the GPU.
- Correlate service latency with tensor/SM, DRAM, PCIe, NVLink, power, and queue metrics.
- Change one lever, replay the same trace, and keep only measured improvements.