A healthy accelerator does not imply a healthy node
A GPU job can compute correctly, report normal device memory and still be terminated because the node is short of host RAM, filesystem space or inodes. Kubelet monitors node-level eviction signals and can end pods to reclaim resources. That mechanism protects the node; it does not judge whether the CUDA kernels were healthy. Treat GPU telemetry as one part of the incident record, not as proof that Kubernetes had no reason to act.
Separate three outcomes immediately. A scheduler rejection leaves the pod Pending. A container OOM kill happens when memory enforcement kills a process and may be followed by a restart according to restartPolicy. A node-pressure eviction sets the selected pod to Failed and terminates it. These paths have different evidence and remedies. Repeatedly increasing device memory or changing GPU type does nothing for a nodefs.available threshold.
Account for the host resources around the job
Host memory can include Python processes, data-loader workers, prefetched batches, pinned transfer buffers, communication libraries, page cache and tmpfs-backed emptyDir volumes. Local ephemeral storage includes writable container layers, node-level container logs and disk-backed emptyDir data. Image storage may share the root filesystem or use a separate image filesystem, which changes the signal kubelet observes and what it attempts to reclaim.
Measure phases separately. Image pull and model staging stress disk before computation begins. Data-loader fan-out can raise host RAM after warm-up. Checkpoint creation can temporarily require both a new file and the previous checkpoint. Verbose logs may grow throughout a multi-day run. Requests based only on the steady training step systematically miss these peaks. Include sidecars and init containers because they contribute to the pod's resource behavior and node occupancy.
Requests influence both placement and eviction risk
For memory pressure, kubelet considers whether a pod's usage exceeds its requests, pod priority and usage relative to requests when ranking candidates. BestEffort or Burstable pods consuming above requests tend to be considered before pods whose usage stays within requests, subject to priority. Guaranteed QoS requires CPU and memory requests and limits for every container to be equal. Even Guaranteed pods are not immortal when the node must protect itself.
Ephemeral-storage pressure differs. Kubernetes documentation notes that QoS classification does not govern ephemeral-storage eviction in the same way. A pod can also be evicted for exceeding its own local ephemeral-storage limit. Therefore set explicit requests from measured use for scheduling and set limits only with a clear failure policy. Priority may change victim order but cannot manufacture RAM, bytes or inodes, and excessive priority can simply move the outage to another workload.
Use an evidence-first diagnostic sequence
First, preserve the failed pod object and events: reason, message, node name, timestamps and container termination states. Second, inspect the node's MemoryPressure, DiskPressure and PIDPressure conditions around that time. Third, obtain kubelet and runtime logs plus metrics for memory.available, nodefs, imagefs or containerfs capacity and inodes as applicable. Fourth, compare every container's usage with requests and limits. Fifth, identify concurrent image pulls, log growth, emptyDir use, checkpoints and node daemons.
Do not begin with a recreated replacement pod; its status cannot explain the previous instance. Correlate clocks and retain events because cluster event histories are often short-lived. If the evidence says Evicted with a memory-pressure message, investigate node and pod host memory. If the container state says OOMKilled, inspect its memory limit and process use. If FailedScheduling appears, return to capacity and constraints. If evidence is missing, label the cause unknown rather than inferring it from GPU utilization.
| Evidence | Classification | Primary next check |
|---|---|---|
| Pod reason Evicted; node MemoryPressure | Node-pressure eviction | memory.available and request-relative usage |
| Container reason OOMKilled | Container memory enforcement or system OOM | Limits, kernel and kubelet logs |
| Pod Pending; FailedScheduling | Placement failure | Allocatable versus requested resources |
| Eviction message cites ephemeral storage | Disk use or limit | Logs, layers, emptyDir and filesystem signals |
Worked example: reserve for the threshold
Assume an illustrative node has 512 GiB physical RAM. The operator reserves 32 GiB for the operating system and Kubernetes daemons, then adopts a separate 16 GiB safety margin. The resulting local planning ceiling for pod requests is 512 minus 32 minus 16, or 464 GiB. Seven jobs requesting 64 GiB each total 448 GiB and leave 16 GiB below that ceiling. An eighth cannot be admitted under this policy even if its GPU slot is free. The scheduler enforces this ceiling only if reservations, allocatable capacity or admission controls actually reflect it.
Suppose each job actually peaks at 72 GiB. Seven jobs then use 504 GiB, which is 56 GiB above their combined 448 GiB of requests. Adding the illustrative 32 GiB system reservation produces 536 GiB of demand, 24 GiB beyond physical RAM, before considering other variation. The decision rule is: sum of pod requests must stay within an enforced planning ceiling aligned with system reservations and eviction policy, while each job request must cover the chosen observed percentile plus burst allowance. These numbers are illustrative arithmetic, not a universal configuration. Actual kernel, daemon and workload behavior must be measured.
Prevent recurrence without hiding pressure
Set CPU, memory and ephemeral-storage requests for every container from phase-aware observations. Bound data-loader concurrency, cache size, log retention and checkpoint generations. Put durable checkpoints on appropriate persistent storage and make jobs restartable; local ephemeral storage has no durability guarantee after node failure. Alert on distance to eviction thresholds, inode exhaustion, pod usage above request and unexpected writable-layer growth. Validate that kubelet reservations and eviction policy are consistent with node allocatable capacity.
Match GPU nodes to the measured host-memory and storage envelope instead of choosing on accelerator count alone. Accurate requests reduce surprise but cannot prevent hardware failure, leaks or sudden shared-node pressure. Revisit the envelope whenever worker count, input pipeline, logging or checkpoint format changes. Verify that the replacement pod has enough cluster capacity to restart elsewhere; recovery logic is ineffective when every eligible node is already full. Include node daemons in forecasts because their growth can consume the margin even when application demand is unchanged. Limits can bound damage but may terminate the job sooner. Keep checkpoints, capture evidence automatically, and test recovery so a protective eviction becomes a controlled interruption rather than lost computation.