Two failures share one misleading word
GPU memory fragmentation is a runtime allocation problem inside a device context. An allocator may have free bytes, yet no suitable block for a requested allocation, or it may reserve segments whose inactive pieces cannot be reused efficiently. Scheduler fragmentation is a placement problem outside the process. The cluster may have enough aggregate GPUs, slices, CPU or RAM, yet no single node has the exact combination required by the next pod.
The symptom can look similar: capacity exists on a dashboard and work cannot start. The boundary is decisive. If the pod is Running and the framework reports an allocation failure, investigate device memory and allocator state. If the pod is Pending and has no application logs, investigate node-level allocatable resources and scheduling predicates. Do not restart an allocator to cure a placement constraint, and do not reconfigure nodes merely because one process retained cached blocks.
Measure allocator state inside the process
A framework allocator distinguishes memory allocated to live tensors from memory reserved from CUDA for reuse. In PyTorch, a memory snapshot exposes segments returned by cudaMalloc and blocks within those segments. The snapshot viewer can show allocation history and out-of-memory events. Compare allocated bytes, reserved bytes, inactive blocks and the size of the failed request at the same workload phase. Device-wide tools alone cannot attribute every byte to PyTorch, because CUDA libraries and other allocators may allocate directly.
Use a repeatable sequence: reproduce with fixed model, shapes and concurrency; record allocator history before the failure; capture a snapshot at the failure boundary; inspect whether substantial inactive space is split across blocks smaller than the request; then compare device usage reported outside the framework. A large reserved-minus-allocated gap is a clue, not proof of fragmentation. It can reflect reusable cache or non-framework allocations, so the block map and failed allocation size matter.
Measure stranded capacity by node and resource shape
For scheduler fragmentation, build a per-node matrix from allocatable minus already requested resources. Include every dimension that filters placement: the exact extended-resource name, CPU, memory, ephemeral storage, labels, taints and topology. Aggregate free GPUs are not a fit metric. Four nodes with one free GPU each cannot admit a two-GPU pod. Likewise, a node with two free GPUs but only 20 GiB unrequested host RAM cannot admit a pod asking for two GPUs and 64 GiB.
Define stranded capacity relative to a demand shape. For request vector q, count free devices on nodes that fail at least one other required dimension, and separately count nodes that pass all filters but lack the device count. This measurement changes with q. There is no universal stranded-GPU percentage because a one-GPU inference replica and an eight-GPU training worker see different feasible sets. Scheduler events provide the first classification; the node matrix explains the arithmetic.
| Observation | Likely layer | Next evidence |
|---|---|---|
| Running process fails an allocation | Allocator or device memory | Memory snapshot and failed block size |
| Pod remains Pending | Scheduler placement | FailedScheduling events and per-node requests |
| MIG capacity exists under another profile | Device partition shape | Advertised profile resources and placement layout |
Worked example: 12 GiB free is not one fact
Illustrative allocator case: a snapshot accounts for 40 GiB reserved in allocator segments, including 28 GiB in live allocations and 12 GiB in inactive blocks. A new 8 GiB request fails because the largest reusable block is 5 GiB and device headroom cannot supply a new segment. In this deliberately simplified snapshot, 40 minus 28 equals 12 GiB, but reserved minus allocated is not universally identical to inactive space because blocks awaiting stream completion and allocator metadata can affect the accounting. The relevant evidence is the block states and the largest suitable block, not the subtraction alone. Shape stabilization, shorter-lived references or allocator configuration may help after snapshot evidence confirms the pattern.
Illustrative scheduler case: two nodes each expose four GPUs. Node A has one free GPU and 100 GiB free host RAM; node B has three free GPUs and 20 GiB free host RAM. A pod asks for two GPUs and 48 GiB RAM. The cluster totals four free GPUs and 120 GiB RAM, but neither node fits both dimensions. These figures are illustrative, not measured performance. The remedy is placement or workload reshaping, not clearing a framework cache.
Apply remedies at the layer that owns the waste
For allocator fragmentation, first remove unintended tensor retention and unbounded shape variation. Align batching into a manageable set of shapes, release references at safe lifecycle boundaries, and use framework allocator controls only after profiling. PyTorch documents max_split_size_mb as a last-resort option for workloads with inactive split blocks; configuration can trade fragmentation against performance. Emptying a cache may release wholly unused cached memory, but it cannot free live tensors and is not a general cure.
For scheduler fragmentation, make requests reflect measured needs, choose replica sizes that match node shapes, and configure scheduler scoring when consolidation is the objective. Kubernetes NodeResourcesFit supports MostAllocated and RequestedToCapacityRatio, including weights for extended resources. For partitioned devices, standardize a small profile catalog when practical. NVIDIA MIG profiles combine fixed memory and compute slices, so free slices may not form a requested profile. Reconfiguration can disrupt workloads and must follow the platform's safe process.
Use a binary triage rule, then accept the limits
Use this decision rule: no bound pod plus FailedScheduling means placement first; a bound Running pod plus a framework allocation error means allocator first; a bound pod that cannot see its assigned device means device-plugin or runtime integration first. Stop after each layer produces evidence. Track allocator failures per workload phase and scheduler admission failures per request shape. A useful scheduler KPI is feasible nodes divided by eligible nodes for each important replica shape, not fleet-wide average utilization.
ClusterBid can help match capacity to those measured request shapes, but neither marketplace selection nor bin packing eliminates fragmentation. Dynamic workloads change shapes, failures remove nodes, and strict spreading intentionally sacrifices packing for availability. Allocator snapshots also impose observation overhead and may miss allocations made outside the framework. Repeat the measurement after model, batch policy, partition profile or node mix changes. The honest goal is bounded waste with explainable evidence, not a permanently perfect utilization percentage.