A GPU is only one coordinate in the fit test
A pod that requests a GPU does not reserve a usable server by itself. The Kubernetes scheduler must find one node that simultaneously satisfies every container request in the pod: the extended GPU resource, CPU, memory, ephemeral storage when requested, selectors, taints, affinity and any topology rules. A node with a free GPU can therefore be an invalid destination because its allocatable RAM has already been requested. Conversely, a node with abundant RAM is useless if its device plugin advertises no healthy GPU.
This distinction explains a common misleading dashboard. Hardware telemetry may show an idle accelerator while the pod remains Pending. Scheduling is based primarily on declared requests and node allocatable capacity, not the instantaneous CPU or RAM utilization graph. Treat the pod as a vector such as one GPU, 6 CPU cores, 48 GiB memory and 80 GiB ephemeral storage. Admission requires a node with room in every coordinate at the same time.
| Signal | What it establishes | What it does not establish |
|---|---|---|
| Free GPU in telemetry | The device appears idle | The pod fits CPU, RAM and policy constraints |
| nvidia.com/gpu in node capacity | The device plugin advertised devices | All advertised devices are allocatable and unclaimed |
| Low memory utilization | Current consumption is low | Requested memory leaves schedulable headroom |
Verify the extended-resource contract first
Kubernetes relies on a vendor device plugin to register a resource name such as nvidia.com/gpu with each kubelet. The kubelet reports healthy devices in node status, and pods request an integer quantity. Extended resources cannot be overcommitted. For GPUs, Kubernetes documents that the resource normally appears in limits; if both requests and limits are given, the values must match. A request for one device is not a request for some arbitrary fraction of its memory.
Begin diagnosis with node capacity and allocatable values, then inspect the pod specification exactly as admitted. Confirm that the driver and device-plugin DaemonSet are healthy on the candidate nodes, and compare the resource name character for character. If allocatable is zero or absent, changing pod CPU will not help. If the GPU resource exists but is fully allocated, reducing a RAM request will not create a GPU. This staged check prevents unrelated tuning.
Measure the host-side work around each accelerator
CPU feeds tokenization, decompression, data loading, request routing and framework control paths. Host RAM holds processes, page cache, pinned buffers, prefetched batches and sometimes model staging copies. Scratch storage absorbs image layers, logs, compilation caches, spilled artifacts and emptyDir data. These resources can peak at different moments, so a single steady-state sample is weak evidence. Capture per-container usage across startup, warm-up, representative load and shutdown or checkpointing.
Set requests from a high percentile of repeatable observations plus explicit headroom for known bursts, rather than copying node capacity into every pod. Keep limits deliberate. A tight CPU limit can cause throttling even when the GPU is idle, while a memory limit can produce a container OOM kill. Local ephemeral-storage limits can trigger pod eviction when writable layers, logs and disk-backed emptyDir usage exceed the limit. Memory-backed tmpfs emptyDir counts as memory, not local ephemeral storage.
Worked example: test the whole envelope
Consider an illustrative node with 8 GPUs, 64 allocatable CPU cores and 480 GiB allocatable memory after system reservations. Existing pods request 6 GPUs, 44 cores and 360 GiB. A new replica requests 1 GPU, 12 cores and 96 GiB. After placement the totals would be 7 GPUs, 56 cores and 456 GiB, so it fits. A second identical replica would produce 8 GPUs, 68 cores and 552 GiB. It is Pending because both CPU and memory exceed allocatable, even though the eighth GPU looked available before that attempt.
The measurable rule is simple: for every candidate node and every resource r, existing requested r plus incoming requested r must not exceed allocatable r, and all non-resource predicates must also pass. Derive the incoming pod request with the rules implemented by the cluster version: account for concurrent application containers and sidecars, the effective request of init containers, and any declared pod overhead rather than reading only the main GPU container. These numbers are illustrative capacity arithmetic, not a benchmark. Real daemon overhead and workload peaks must be measured in the target cluster.
Pending and evicted require different evidence
For Pending pods, read scheduling events before looking at application logs. A FailedScheduling event can enumerate insufficient CPU, insufficient memory, unavailable extended resources, untolerated taints or affinity conflicts. Then compare kubectl describe node output with requests already assigned. Preemption may still fail when no single lower-priority set frees the complete multidimensional envelope, or when policy constraints exclude the remaining nodes.
For a pod that ran and later disappeared, inspect pod status, termination reason, node conditions and events. A node-pressure eviction is initiated by kubelet when signals such as memory.available or filesystem space cross configured thresholds. The kubelet ranks pods using whether usage exceeds requests, priority and usage relative to requests. That is different from a container-level OOM kill and different again from a scheduler rejection. Preserve events promptly because retention is finite, and correlate them with node memory, filesystem and inode telemetry.
Turn measurements into an admission rule
Maintain one resource-envelope record per workload version: startup peak, steady-state percentile, exceptional phase, request, limit and evidence window for CPU, memory and ephemeral storage, plus the exact GPU resource name and count. Reject promotion when any request is below the chosen observed percentile plus headroom, or when a representative node cannot host the intended replicas after DaemonSet and system reservations. Also reject configurations that depend on average free capacity spread across several nodes, because one pod cannot combine fragments from different machines.
Compare candidate nodes against that measured vector rather than reducing the requirement to GPU model alone. Requests express admission intent, not performance assurance. A schedulable pod can still suffer CPU throttling, storage latency, NUMA effects or application leaks. Recalibrate after image, model, batch-size, worker-count or logging changes, and validate under realistic concurrency before treating the envelope as stable.