An inexpensive GPU-hour can produce an expensive answer
The denominator must be the outcome that passed its performance, quality, and reliability contract. Include all resources and failed work in the numerator.
| Workload | Useful denominator | Cost that belongs in the numerator |
|---|---|---|
| Online inference | Successful requests or output tokens inside TTFT/ITL/error SLO | Warm/idle replicas, nodes, CPU/RAM, gateway, storage, network, failures and retries |
| Batch inference | Completed jobs/tokens before deadline | All attempts, queue delay, interruption/replay, storage and egress |
| Training | Accepted optimizer steps or successful run | Every GPU attempt, checkpoint I/O, replay, data pipeline, fabric, control plane |
| Media / rendering | Frames/streams/jobs at quality target | Codec/CPU stages, memory transfer, licensing, failed frames, energy |
| HPC | Completed solve at required tolerance | Porting, node/fabric, checkpoint/restart, energy and scheduler wait |
effective $ / useful GPU-hour = fully loaded cost ÷ useful GPU-hours$ / 1M successful output tokens = deployment $/h × 1,000,000 ÷ successful output tokens/hMake two quotes comparable
Replace the illustrative inputs with the complete node price and the measured SLO-passing output of the same workload.
Illustrative inputs · replace with your quote
Comparable output
$5.98/ 1M successful output tokens
- Fully loaded deployment
- $14.00 / hour
- Commit break-even
- 60% eligible utilization
“Successful” means the output met the latency, quality, and error-rate target. Failed and retried work belongs in the numerator, not the token count.
Commit the floor, burst the uncertainty, interrupt only recoverable work
| Model | Best fit | Economic trap | Required control |
|---|---|---|---|
| On-demand | Benchmarks, launches, uncertain demand, short bursts | Highest headline rate | Expiry, autoscaling, and right-sized pools |
| Commitment / reservation discount | Conservative continuously used baseline | Discounted idle time; commitment may not reserve capacity | Break-even utilization and separate capacity assurance |
| Spot / preemptible | Checkpointable training, batch, redundant stateless replicas | Eviction, unavailable SKU, replay and no SLA | Durable checkpoints, broad placement, fallback and tested resume |
| Capacity reservation / block | Deadline or launch capacity that must exist | Paying for unused reserved supply | Explicit start/end, owner and release policy |
| Owned / colocation | Stable high utilization with power/cooling and operations capability | Stranded capital, facility lead time, maintenance and obsolescence | Lifecycle, spares, energy, utilization and residual-value model |
commitment break-even utilization = committed effective $/h ÷ on-demand $/hspot effective rate = spot $/h ÷ useful-work goodput fractionTreat price and availability as timestamped data
Absolute cloud prices, regions, quota, commitment terms, lifecycle, software matrices, and preliminary products change too quickly for evergreen prose.
- Store provider, region/zone, exact instance shape and GPU count, OS, currency, purchase model, term, source URL, and
verifiedAt. - Separate discount from capacity assurance: some commitments lower bills without reserving the GPU.
- Record minimum quantity, utilization restrictions, data-egress and control-plane charges, and early-exit or transfer rules.
- Refresh official price APIs or signed quotes before every material procurement decision.
- Keep vendor benchmark and TCO claims with the full test configuration and date; never convert them into universal multipliers.