Thermal Throttling During Long GPU Jobs: What to Record

Temperature shows thermal state, not the reason a clock changed. A GPU can operate at a steady, relatively warm temperature without thermal limiting, and clocks can fall because the workload is idle, power-limited, synchronized with another device, or constrained by an application clock setting. Treat a temperature threshold crossed in isolation as an observation that deserves context, not a diagnosis.

Strong evidence of thermal throttling requires aligned signals: useful work continues, clocks decline or performance degrades, and a vendor-reported thermal clock-event reason becomes active or a cumulative thermal-violation duration increases during the same interval. NVIDIA DCGM exposes separate clock-event fields for software thermal slowdown, hardware thermal slowdown, software power cap, idle, and other reasons, although field support varies by device. That distinction prevents a hot-looking chart from becoming an unsupported root-cause claim.

Sample GPU temperature, memory temperature where available, thermal margin, graphics and memory clocks, board power, utilization, clock-event reasons, and cumulative thermal-violation time. Add job progress such as steps per second, tokens per second, batch completion, or phase markers. Record node, device identifier, driver, firmware, software revision, configured power limit, fan or cooling policy exposed to the operator, and ambient or inlet temperature if the facility provides it.

Use a consistent clock and cadence across telemetry and application logs. A one-second cadence may expose sustained changes for many long jobs, but the appropriate interval depends on phase length and storage budget. Preserve raw data, not only dashboards. Aggregated five-minute averages can hide a short limiter event, while overly sparse application logging makes it impossible to tell whether a clock drop occurred during compute, checkpointing, evaluation, or input waiting.

Long jobs rarely present constant demand. Initialization, compilation, data loading, forward and backward passes, communication, evaluation, and checkpointing exercise different components. Insert phase markers or derive them from application events, then compare like phases. Falling clocks during a checkpoint may be normal because the GPU has little work; the same fall during a stable compute phase may require investigation.

Choose a baseline segment after warm-up when throughput, utilization, and clocks are stable. Compare later instances of that same phase. Also compare peer GPUs performing equivalent work, while recognizing that rank roles and data skew can differ. The purpose is not to demand identical temperature or clock values. It is to identify a repeatable change that correlates with a specific limit reason and loss of completed work.

First verify the application symptom: longer step time or lower completed-work rate during the same phase. Second check whether GPU utilization indicates continuing demand. Third look for a simultaneous clock reduction. Fourth inspect clock-event reasons and thermal-violation counters. Fifth compare temperature margin and temperatures around the transition. Sixth exclude power-cap events, idle periods, synchronization, errors, and host or input bottlenecks. Finally repeat the phase or job under comparable conditions.

Use this decision rule: label an interval confirmed thermal limiting only when a supported vendor thermal event field becomes active, or its cumulative duration increases, while demanded compute continues and clocks or throughput fall. Label it suspected when the correlated performance and thermal-margin pattern repeats but the platform does not expose limiter reasons. Label it not demonstrated when temperature rises without a thermal event or performance change. This vocabulary keeps the incident record useful even with incomplete telemetry.

Imagine an illustrative six-hour training job. From hour one to hour three, a repeated compute phase averages 2.0 seconds per step, the reported graphics clock is stable, and no thermal event is active. After hour four, temperature is 6 C higher, but step time remains 2.0 seconds and the thermal event counter does not change. That is warm operation, not evidence of throttling.

At hour five, the same phase rises to 2.3 seconds per step, a 15 percent increase because 0.3 divided by 2.0 equals 0.15. GPU utilization remains high, graphics clocks fall, and the thermal event reason becomes active for matching timestamps. That interval supports thermal limiting as the immediate mechanism. It still does not prove the underlying fault is a cold plate, airflow, coolant loop, sensor, or facility condition. Physical root cause needs authorized inspection and corroborating system data.

Repeatability raises confidence. Check whether the event occurs at a similar elapsed time, phase, thermal margin, or facility condition across multiple runs. Compare devices in the same node and equivalent nodes. A single affected GPU suggests a local path or sensor possibility; a rack-wide pattern suggests a shared environmental or control possibility. These are diagnostic directions, not conclusions.

Recovery behavior also matters. Note whether clocks and throughput return after a lower-demand phase, whether thermal event flags clear, and whether violation counters continue to accumulate. Do not modify cooling hardware or bypass safety controls to test a theory. Escalate with the aligned trace, system identifiers, and reproduction window so the provider or authorized facilities team can investigate within documented procedures.

Create an alert that requires more than temperature. One illustrative rule is: during a declared compute phase, alert when a thermal clock-event reason is active for 60 cumulative seconds and median step time is at least 10 percent worse than that job's stable baseline. Tune duration and regression to workload sensitivity and sampling cadence. A separate warning can track shrinking thermal margin without declaring throttling.

The limitation is observability. Virtualized or managed environments may hide event reasons, memory temperature, fan state, or facility sensors. Counters may reset with a driver reload, and clock readings are samples rather than a continuous record. When direct evidence is unavailable, report the case as suspected, retain the application-level trace, and ask the provider for synchronized limiter telemetry. Precision in the label is more valuable than certainty manufactured from one hot sensor.

NVIDIA DCGM Field Identifiers

NVIDIA DCGM Field Constants

NVIDIA DCGM Health Feature Overview

GB300 NVL72 Rack-Scale Buyers Guide 2026: Power, Cooling, and Real Cost

Academic GPU Compute Grants: NSF, NIH, and University AI Funding in 2026