No single utilization gauge explains the service
A useful dashboard connects user experience to the engine queue, GPU/fabric counters, and cluster placement.
Service
Request rate, errors, cancellations, input/output distributions, p50/p95/p99 TTFT/ITL/E2E, goodput, cold starts, cost per useful unit.
Engine
Running/waiting/preempted requests, queue time, prefill/decode time, batch tokens/sequences, KV capacity/occupancy, cache hits/evictions, speculation.
GPU + fabric
HBM used, SM/tensor and DRAM activity, PCIe/NVLink traffic, power/energy, clocks/throttling, temperature, XID/ECC and link health.
Cluster
Allocated/available GPUs by SKU/profile, unschedulable reasons, queue admission, quota/preemption, node startup, topology, restarts and fragmentation.
Use correlated signals to narrow the bottleneck
| Symptom | Likely direction | First proof |
|---|---|---|
| Queue high; GPU activity low | CPU/tokenization, frontend, scheduler, network, tiny batches, kernel fallback | Profile queue stages and host; inspect batch/token size and kernel path |
| DRAM high; tensor low | Decode memory-bound | Sweep batching; test weight/KV precision and speculation separately |
| Tensor high; TTFT long | Prefill compute or queue-bound | Split queue from prefill time; test cache/chunking/faster compute |
| KV near full; preemption/swap; tail spikes | Unsustainable live-token concurrency | Lower admission/context; increase/quantize KV; inspect block use |
| Link traffic high; scaling weak | Excess TP/EP communication or poor placement | Compare smaller shard count and verified topology |
| Average good; p99 ITL bad | Prefill interference, bursts, stalls, collectives or cold replicas | Long-prompt probes and phase/replica correlation |
| High prefix potential; low hit tokens | Routing, unstable tokens, eviction, small cache | Log token-prefix identity, route affinity and eviction |
| High GPU activity; low training progress | Data stalls, stragglers, checkpoint/replay, rejected/overflowed steps | Accepted-step goodput and per-rank timeline |
Benchmark recovery before the incident
- Remove one replica and verify load shedding, queue growth, tail latency, and recovery capacity.
- Replace one GPU node; time provisioning, image pull, weight download, compile/graph capture, readiness, and cache warming.
- Roll the engine/model while streams remain open; verify draining, cancellation, retries, and version isolation.
- Lose prefix/KV state and measure the cold TTFT and capacity impact.
- Inject checkpoint/object-store failure and resume from the last known durable state.
- Exercise rank, collective, NIC, and peer-link failure for multi-GPU work.
- Overload the gateway and engine; verify admission, bounded queues, timeouts, and honest error codes.
- Test loss of a region/zone or capacity pool if the availability target requires it.
expected waste = checkpoint overhead + failure probability × replay since last durable checkpointThe minimum GPU service runbook
Identity
Owner, model/app and engine versions, container digest, hardware pool, topology, region, data class, dependencies and current rollback target.
SLO + capacity
Traffic envelope, latency/error/quality targets, per-replica goodput, minimum warm capacity, headroom, overload policy and quota.
Signals
Service, queue, KV, batch, GPU/fabric, scheduler and cost dashboards with alert thresholds and measurement boundaries.
Actions
Scale, shed load, drain, restart, re-place, switch pool/model/precision, disable an optimization, restore checkpoint and rotate cache salt.
Change
Compatibility matrix, benchmark gate, canary/rollback, cache/cold-start effect, rollout overlap, data migration and evidence retention.
Economics
Cost owner, useful unit, purchase mix, commitment utilization, idle expiry, Spot recovery, price verification date and variance alert.