A high-water mark is not a leak
GPU memory can rise for healthy reasons: larger batches, longer sequence lengths, model loading, graph capture, workspace selection, and cache growth. An alert on a single percentage threshold therefore pages on normal workload shape. A leak is better framed as memory that remains attributable after the work that justified it has ended, or growth that repeatedly reduces allocatable headroom for equivalent work.
Start with two signals, not one. Device telemetry supplies framebuffer used and free memory, while the serving runtime supplies active requests, queued work, cache occupancy, and allocation failures. NVIDIA's DCGM exporter documents framebuffer-used and framebuffer-free gauges, but also notes that configured fields are not guaranteed to appear on every system. Validate the actual metrics endpoint before depending on a field.
Build baselines by workload shape
Compare like with like. Partition observations by model version, worker configuration, batch-size band, input-length band, and concurrency band. For each bucket, record memory before the work, peak memory, memory after a recovery interval, and the number of successful requests. The relevant baseline is a distribution, not the fleet-wide average, because a long-context worker and an image classifier have different legitimate plateaus.
Use a rolling reference that excludes deployments and known incidents. One practical rule is to warn when the post-recovery level exceeds the bucket's recent p95 by both 10 percent and 2 GiB for 30 minutes. Those values are illustrative. The relative condition catches proportional change, the absolute condition ignores tiny movement, and the persistence window prevents a brief large request from becoming an urgent page.
Separate cache growth from retained objects
Caching allocators deliberately retain blocks so later allocations avoid synchronization and device allocation overhead. Framework-reported allocated memory can therefore fall while device-level used memory stays high. A healthy cache usually reaches a plateau for repeated workload shapes and remains reusable. Suspected retention instead produces a rising floor, a shrinking largest reusable block where the allocator exposes that statistic, or more failed allocations at the same concurrency. Device-level totals alone do not reveal block contiguity or ownership.
Run a controlled diagnostic sequence on one worker. Stop new admissions, wait for active requests and queues to reach zero, record framework allocated and reserved memory, trigger only the runtime's documented cache-release operation if operationally safe, then observe device memory. A drop after cache release suggests reusable reservation; no drop may indicate live references, non-framework allocations, another process, or driver-managed memory. It does not prove a leak by itself.
Page on consequence and persistence
Use separate warning and paging conditions. A warning can track abnormal residual growth for investigation. A page should require customer or capacity risk, such as a recent out-of-memory allocation failure, repeated request failures correlated with low free memory, or projected exhaustion before an operator can respond. Prometheus supports a for clause that requires an expression to stay active before firing, which is useful for persistence rather than spike sensitivity.
An illustrative page rule is: fire when free framebuffer remains below the worker's measured safe-admission reserve for 10 minutes and either allocation failures increase or request errors exceed their error-budget threshold. The safe reserve should be measured from the largest admitted request class plus runtime overhead, not copied across models. Keep a separate immediate alert for critical device health events; memory pressure and hardware faults have different runbooks.
Use slopes only inside stable windows
A positive memory slope is meaningful only when demand is stable. Compute it during windows where model version, worker count, concurrency band, and request-size mix remain within defined ranges. If memory rises 300 MiB every 15 minutes while completed requests and input lengths are steady, the signal deserves inspection. If concurrency also climbs, the same slope may simply describe useful cache population.
Worked example: an illustrative worker idles at 18 GiB after warmup. Four equivalent 20 minute traffic blocks end at 19, 20, 21, and 22 GiB after a five minute recovery interval. That is a 1 GiB step per block and a 4 GiB residual increase from baseline. If allocation failures begin at 23 GiB, paging at the raw 20 GiB point would be early; warning on the repeated residual steps and paging on failed allocations plus low reserve is more precise.
Correlate before restarting
When an alert fires, first confirm whether the growth is isolated to one worker or follows a model release across the fleet. Next compare active requests, queue depth, request lengths, cache metrics, framework allocated and reserved bytes, other GPU processes, and allocation failures. Then drain one affected worker and observe its recovery floor. Finally capture a memory snapshot or allocator history if the framework supports it before restarting destroys the evidence.
Automatic restart is reasonable only as a bounded recovery action. Set a rule such as one automated drain and restart per worker per six hours, followed by escalation if the residual pattern returns. Otherwise a crash loop can erase diagnostics while reducing serving capacity. The limitation is metric resolution: device totals cannot identify which tensor, process, or cache owns the bytes, so root cause still requires runtime-level evidence.
Test alerts with replayable patterns
Validate alert logic with replayable traces rather than waiting for production. Include a single large request, a staircase of increasing sequence lengths, stable repeated batches, cache warmup to a plateau, an intentionally retained allocation in a test build, and an induced allocation failure. Confirm that the peak-only cases create dashboards or warnings but not pages, while persistent residual growth combined with consequence crosses the paging rule.
Track alert precision as the share of pages requiring immediate operator action, plus time from first harmful residual growth to detection. Review thresholds when model shapes change. ClusterBid can match capacity options to measured memory headroom when a model genuinely needs larger devices, but additional memory should not be used to conceal unbounded retention or a missing admission limit.