Five gates prevent an expensive shortcut
A GPU decision is valid only after the workload passes all five gates. A famous SKU can fail on fit, software, fabric, facility, or unit cost.
Weights + state + KV/activations + workspace + headroom
Required precision, kernels, codecs, and software support
TTFT, token cadence, throughput, quality, and reliability
One GPU, linked node, or multi-node fabric - and the physical envelope
Cost per successful unit at the SLO, including idle and failed work
Workload envelope
Model or kernel, precision, input/output shape, batch and concurrency, context length, data movement, duration, and failure tolerance.
Service contract
p50/p95/p99 latency, useful throughput, quality gates, deadline, availability, recovery point, and regional or data-governance limits.
Operating envelope
Framework and engine, driver matrix, rack or cloud shape, quota, interconnect, power, cooling, team skill, and purchase term.
Build a benchmark shortlist
The worksheet deliberately stops at a candidate lane. It does not manufacture a precision recommendation from a few dropdowns.
Translate constraints into a test plan
Use the estimated resident footprint: weights, KV or activations, engine workspace, and operating headroom - not the checkpoint file alone.
- Topology
- Start with one GPU per replica; add data-parallel replicas only after the single-replica benchmark passes.
- Proof
- Sweep concurrency until p95 TTFT or TPOT fails, then price the highest SLO-compliant point.
- Guardrail
- Optimize p95/p99, not only the single-request best case; queueing and mixed prompt lengths decide production latency.
This is not a benchmark or purchase recommendation. Verify exact SKU, software version, region, power envelope, and production trace.
Write the decision before requesting quotes
A one-page record keeps benchmark results, provider quotes, and architecture tradeoffs comparable.
| Field | Minimum evidence | Reject when |
|---|---|---|
| Workload | Versioned model/kernel, real input and output distributions | Only a model name or parameter count is supplied |
| Memory | Measured peak residency plus rollout/failure headroom | Checkpoint size is treated as total memory |
| Performance | SLO goodput with p95/p99 latency and errors | Only peak FLOPS or unconstrained tokens/s is shown |
| Topology | Exact GPU form factor, links, host, NICs, placement | The quote says only “8× H100” |
| Software | Pinned driver, framework, engine, model and precision matrix | Support is inferred from architecture name |
| Cost | Fully loaded cost per useful output or accepted step | Comparison stops at $/GPU-hour |
| Risk | Capacity, interruption, restart, upgrade, and fallback test | Availability or recovery is assumed |