Different workloads reward different parts of the system
There is no universally best accelerator. The correct shortlist changes when the useful unit changes.
| Workload | First constraints | Candidate lane | Measure |
|---|---|---|---|
| Interactive LLM | Weights + KV fit; p95/p99 TTFT and token cadence | Smallest fast single GPU, then linked node only if required | SLO goodput and $/1M successful output tokens |
| Batch generation | Aggregate memory, throughput, deadline, interruption tolerance | Replica-parallel right-sized GPUs; flexible/Spot capacity | Completed tokens or jobs per dollar before deadline |
| Full training | Parameters, optimizer, gradients, activations, collective topology | SXM/OAM scale-up node, then topology-aware multi-node | Accepted steps/GPU-hour and time-to-result |
| Fine-tuning / LoRA | Base weights, trainable state, activation/checkpoint policy | Often fewer/larger-memory GPUs; prove one-node first | Examples/s or accepted steps per dollar |
| Image / video | Resolution, frames, codec blocks, tensor/VRAM need | L4/L40S for media density; larger HBM parts for model-heavy pipelines | Frames/s, latency, quality, energy/frame |
| Rendering / VDI | Graphics APIs, vGPU licensing, framebuffer, encoders | A10/L4/L40S and certified workstation/data-center stack | Frame time, concurrency, experience SLO |
| HPC | FP64/FP32 ratio, memory traffic, MPI/collectives, solver tolerance | A100/Hopper/Blackwell, AMD Instinct; code-portability gate first | Time-to-solution and energy-to-solution |
Memory state and collective traffic dominate the design
Memory budget
Include parameters, gradients, optimizer state, master weights, activations, communication buffers, compiler workspace, and fragmentation. Activation checkpointing trades extra compute for less memory.
Scaling budget
Data parallelism communicates gradients; tensor parallelism communicates within layers; pipeline parallelism sends activations between stages. The slowest collective can set step time.
Goodput
Track accepted optimizer steps, not merely device activity. Data stalls, compilation, stragglers, failed collectives, checkpointing, and replay can leave a GPU busy without moving the model forward.
Recovery
Checkpoint interval should minimize expected replay plus checkpoint overhead. Test resume from durable storage before using interruptible capacity.
training goodput = accepted optimizer steps ÷ purchased GPU-hoursUse the workload’s native quality and performance contract
Image and video. Separate model execution from decode, resize, encode, and transfer. Low-power cards with fixed-function codecs can beat a larger training GPU on streams per rack watt. For diffusion or video generation, resolution, frame count, attention implementation, batch, and VRAM become first-order.
Rendering and visualization. Confirm graphics API, ray-tracing, framebuffer, encoder, vGPU profile, and application certification. AI tensor throughput alone is not a render benchmark.
Scientific/HPC. Record the precision actually used, solver tolerance, memory access pattern, CPU coupling, MPI/collective behavior, and porting effort. A high FP8 rate says little about FP64 time-to-solution.
A benchmark is a contract, not a demo
Two results are comparable only when the workload and measurement boundary are held constant.
- Pin model or application revision, tokenizer, framework, engine/container, driver, firmware, precision, kernels, and compiler flags.
- Record exact server SKU, GPU variant/count, link topology, CPU, RAM, NICs, storage, power cap, and cooling.
- Replay production-shaped inputs: distributions, not averages; arrival burstiness; concurrency; warm and cold states.
- Publish p50/p95/p99 latency, useful throughput, quality/accuracy, errors, energy, and cost together.
- Run a ramp until the SLO fails; do not select the unconstrained saturation point.
- Test one failure, one restart, and one rolling change while traffic or training continues.