Start with two independent observations
An application exit and a GPU Xid are different observations. The process manager may report an exit code, signal, exception, or out-of-memory kill. The NVIDIA driver writes Xid reports to the operating system kernel log or event log. They can occur near each other without proving which caused which. NVIDIA states that an Xid can indicate a hardware problem, NVIDIA software problem, or user application problem, so the code is a starting point rather than a universal root-cause label.
Preserve both timelines before restarting or rotating logs. Record time with timezone, host identity, workload and container identity, GPU UUID and PCI bus address, driver version, application build, model or job identifier, and the first visible symptom. Time synchronization matters: a 20-second clock offset can reverse the apparent order between an orchestrator event and a host log entry.
Collect authoritative driver evidence first
On Linux, search the kernel journal or configured kernel log for NVRM Xid messages across a window before and after the failure. NVIDIA documents Xid records in the operating system kernel log and shows records that identify a PCI device and Xid code; available identity fields can vary with driver and platform. Save the surrounding log, not only the matching line, because PCIe, memory, thermal, process, or kernel events can add necessary context.
Capture nvidia-smi -q output and, where operational policy permits, an NVIDIA bug report while the relevant state still exists. NVIDIA's documented bug-report script collects driver and system information for escalation. Treat that archive as potentially sensitive because it contains host configuration and logs; store and share it under incident-data rules. Do not reboot merely to make collection easier, since a reboot can erase volatile evidence.
Capture the process side independently
From the application layer, preserve exit code, terminating signal, stderr, framework exception, core-dump metadata if enabled, and orchestrator reason. Also capture memory limits, host out-of-memory events, liveness failures, eviction events, and deployment changes. A segmentation fault, Python exception, explicit abort, container memory kill, and readiness-triggered restart all have different first branches even if users saw the same disconnected request.
Identify the last successful GPU operation visible to the application and whether CUDA errors appeared before exit. Asynchronous GPU execution means an error can surface at a later synchronization point than the operation that triggered it. That timing uncertainty is a diagnostic limitation. Avoid asserting that the final logged kernel or API call caused the incident solely because it was last.
Build a correlation matrix across failures
One incident rarely separates hardware, driver, and application causes. Compare recurrence by GPU UUID, PCI address, host, application build, input, and workload placement. If different applications fail on one device and Xids recur on that same UUID, device or host investigation gains priority. If the same deterministic input fails across healthy devices without Xids, application investigation gains priority. These patterns guide testing; they are not proofs.
Use a simple matrix rather than narrative memory. For every failure, mark whether an Xid occurred within the chosen window, whether the application exited, and whether the GPU remained usable afterward. Choose the correlation window from logging delay measurements, perhaps five minutes as an illustrative starting point. Report Xid-with-crash, Xid-without-crash, crash-without-Xid, and neither. The fourth category helps estimate background event rates and monitoring gaps.
Follow a bounded diagnostic sequence
First, freeze the event window and collect kernel, application, orchestrator, and GPU identity evidence. Second, extract the Xid code exactly and consult NVIDIA's Xid documentation or catalog for the applicable GPU and driver generation. Third, check whether the application independently reported a programming or memory error. Fourth, compare recurrence across device, host, build, and input. Fifth, quarantine only the scope justified by evidence, then run an approved health test when the device is no longer serving traffic.
A measurable rule can be: classify the incident as Xid-confirmed only when an authoritative host driver record names the device during the incident window; classify it as application-only only when process evidence exists and the host log is complete for that window with no Xid; otherwise retain unknown or correlated status. This prevents a dashboard alert from becoming stronger evidence than the underlying log. Never use absence from an incomplete journal as proof of absence.
Use diagnostics with their stated limits
NVIDIA DCGM provides health monitoring and active diagnostics covering software access, context creation, memory, PCIe, compute, and other paths depending on the selected test and hardware. Run invasive tests only after draining workloads and according to the platform owner's procedure. Capture the exact test, target entity, configuration, start time, result, and diagnostic messages so a later reviewer knows what was actually exercised.
A passed diagnostic does not erase the earlier Xid, and a failed diagnostic does not by itself establish repair or replacement eligibility. NVIDIA explicitly notes that DCGM diagnostics do not repair faults, replace offline field diagnostics, or determine RMA eligibility. Intermittent problems may not recur during a short test. Preserve the incident bundle and follow the hardware provider's or NVIDIA's escalation process when guidance calls for support.
Avoid universal reset advice
Do not turn an Xid number into a blanket command to reset a GPU, reload a driver, or reboot a host. The supported recovery action can depend on Xid, GPU generation, platform topology, virtualization, fabric state, other active workloads, and driver guidance. A reset may be unsupported, fail because clients remain attached, disrupt neighboring devices, or destroy evidence. Consult the applicable NVIDIA catalog and the system vendor runbook for immediate and investigatory actions.
Automated remediation should therefore use an allowlisted mapping tied to validated hardware and software versions. Unknown codes, repeated events, multi-GPU fabric events, or failed recovery must leave the node drained and escalate. Record every automated action and its result. This is slower than a universal reset rule, but it separates service restoration from root-cause claims and prevents one recovery recipe from being applied to an incompatible platform.
Write the incident verdict carefully
The final incident record should state observed facts, supporting artifacts, diagnostic possibilities, actions taken, and unresolved questions. A sound verdict might say that the application exited 1.2 seconds after a driver-recorded Xid on the assigned GPU and that the same Xid recurred on that UUID under a second workload. It should not leap to defective hardware unless the applicable vendor process supports that conclusion.
Track recurrence per device-hour and per application run, plus crashes without Xids and Xids without crashes. Escalate when the documented vendor action says to, when the same device repeats after approved recovery, or when multiple workloads reproduce the event. The honest endpoint is sometimes unknown. A disciplined unknown, backed by preserved evidence, is more useful than a confident but unsupported label.