A busy GPU is not necessarily producing useful tokens
GPU utilization usually reports activity during a sampling interval, not useful work per request. A device can look busy while processing a long prompt, moving data, executing an inefficient kernel, recomputing evicted state, or serving work that will be discarded. Even streaming multiprocessor activity can include warps waiting on memory. The operational question is therefore not whether the GPU is active, but which request phase consumed the active cycles and what completed work resulted.
Define useful throughput before opening a profiler. For an online text service, track accepted prompt tokens, generated output tokens, completed requests, and successful output tokens per second. Keep cancelled and failed work separate. Pair these counters with time to first token, inter-token latency, queue time, prompt length, output length, batch occupancy, and model revision. One aggregate utilization graph cannot explain which term changed.
Split prefill from decode before blaming kernels
Prefill processes the input sequence and commonly produces no visible output until the first token. Decode then advances sequences token by token. A traffic shift toward longer prompts can raise GPU activity and prompt-token throughput while output-token throughput falls. That is not automatically a regression. It may be the expected cost of a different workload mix. Compare prompt and generation token rates together, and bucket latency by prompt length rather than averaging unlike requests.
A first diagnostic split is mechanical. If time to first token rises while inter-token latency stays stable, inspect queueing and prefill. If time to first token is stable but time per output token rises, inspect decode batching, KV-cache pressure, synchronization, and kernels used during decode. If both rise, check admission pressure, preemption, memory capacity, clock throttling, and cross-device communication before narrowing further.
| Observed change | First possibilities | Correlating measurement |
|---|---|---|
| TTFT up, decode stable | Longer prompts or queueing | Prompt-length buckets and queue time |
| TTFT stable, decode slower | Decode batch or KV-cache pressure | Active sequences, cache use, inter-token latency |
| GPU busy, tensor activity low | Non-tensor kernels or stalls | Tensor, SM, and DRAM activity together |
| All phases slower | Throttling, contention, or synchronization | Clocks, power state, per-rank traces |
Read hardware counters as a set
NVIDIA DCGM distinguishes graphics-engine activity, SM activity, tensor-pipe activity, DRAM activity, occupancy, PCIe traffic, and NVLink traffic. These measures answer different questions. High SM activity with low tensor activity may mean the workload is executing other instructions or waiting. High DRAM activity can support a memory-bound hypothesis, but it does not identify the source. High occupancy alone is not proof of efficiency, and DCGM explicitly cautions that active warps may be waiting on memory.
Align all counters to the same interval and revision. Compare a healthy baseline with the suspect window using the same model, precision, request distribution, and batching policy. Look for relationships rather than a magic threshold: output tokens per GPU-second against tensor activity, prompt tokens per GPU-second against DRAM activity, and inter-token latency against active sequence count. Device averages can hide one slow rank, so retain per-GPU series for multi-GPU serving.
Use a diagnostic sequence that preserves causality
First verify the numerator: count successful generated tokens at the serving boundary and reconcile it with engine counters. Second segment by model, replica, prompt-length band, output-length band, and finish reason. Third determine whether the loss is in queue, prefill, or decode. Fourth compare GPU clock, power, memory, SM, tensor, and DRAM signals. Fifth inspect batch occupancy, KV-cache use, preemption, and per-rank imbalance. Only then capture a short application trace or kernel profile.
Change one factor per confirmation run. Fix the request corpus and arrival schedule, then vary batch limit, sequence limit, precision, or kernel configuration individually. A useful decision rule is to reject an optimization if successful output tokens per GPU-second improve by less than 5 percent while p95 latency worsens by more than 5 percent. Those thresholds are illustrative; set production limits from your service objective and measurement noise before testing.
Worked example: traffic shifts toward long prompts
Assume an illustrative four-GPU replica previously receives 20 requests per second with 500 prompt tokens and 100 output tokens each. Offered prompt volume is 20 multiplied by 500, or 10,000 prompt tokens per second. Offered output volume is 20 multiplied by 100, or 2,000 output tokens per second. After a product change, the rate stays 20 requests per second but prompts average 2,000 tokens. Prompt volume becomes 40,000 tokens per second while requested output volume remains 2,000.
Suppose measured successful output falls to 1,400 tokens per second and GPU activity rises from 75 to 96 percent. That observation does not prove a kernel regression. First bucket time to first token by prompt length. If short-prompt buckets retain prior latency and decode speed while the queue grows behind long prefills, the workload mix is the leading explanation. Confirm by replaying the old corpus at the same arrival rate, then test chunked prefill or admission controls without changing kernels.
Escalate from fleet telemetry to traces
If request mix is unchanged, look for idle gaps hidden inside coarse utilization samples, small or fragmented batches, excessive host-to-device transfers, collective waits, and repeated memory allocation. A serving engine can also spend cycles on requests that later time out. Compare admitted work with completed work and cancellation counts. For tensor-parallel models, one delayed rank can hold the others at synchronization points even when several devices report substantial activity.
DCGM is suited to continuous monitoring, but its interval averages do not identify a kernel or source line. NVIDIA recommends a developer profiler when attribution is needed. Capture a representative, bounded window because profiling can add overhead and hardware counters may conflict. The trace should include CPU threads, CUDA kernels, memory copies, and communication. Match trace request IDs to server events so a long kernel sequence can be tied back to prefill, decode, or cleanup.
State what the evidence cannot prove
Counter values vary by GPU generation, model architecture, precision, and collection configuration. Multiplexed counters may be sampled rather than observed simultaneously. A synthetic replay can remove production effects such as network backpressure, tokenizer load, and cancellations. Conversely, production averages can conceal rare but important tail behavior. Treat every suspected bottleneck as a hypothesis until the request-level and hardware-level evidence move together in a controlled comparison.
Close the incident only when the useful-work metric recovers, not when utilization falls. Require successful output tokens per GPU-second, p95 time to first token, p95 inter-token latency, and error rate to meet their declared bands for at least two representative traffic windows. Record the workload distribution with the result. Without prompt and output length context, the same dashboard can be misread during the next traffic change.