Pack a measured demand shape, not a model label
An inference replica consumes a joint shape: device count or partition profile, device memory, CPU, host RAM and sometimes local storage. Its useful capacity is also conditional on batch policy, sequence lengths, concurrency and latency objective. Two replicas serving the same model can have different shapes when one targets interactive tail latency and another absorbs asynchronous batches. Inventory those shapes before selecting a packing policy.
Record peak device memory after warm-up, host-memory peak, requested CPU, sustainable requests or tokens per second at the accepted latency percentile, and restart time. Separate fixed model memory from load-sensitive cache or workspace. Use representative request distributions rather than one maximum-length input or a synthetic average. The scheduler sees declared resources, while the queue sees delivered service; a plan must satisfy both systems.
Choose device granularity before replica count
Whole-device allocation is operationally simple and provides a clear failure boundary, but a small model may leave memory and compute unused. Hardware partitioning such as NVIDIA MIG exposes fixed profiles with dedicated memory resources and compute slices. Time-sharing can raise apparent occupancy but does not create the same isolation or fixed memory partition. The right granularity depends on whether the replica is constrained by memory capacity, compute, bandwidth, latency variance or engine requirements.
Reject a device shape if the measured peak plus safety allowance exceeds its usable memory, or if load testing misses the queue objective. Among passing shapes, compare useful service capacity per schedulable unit and the residual profile space each placement leaves. A smaller profile is not automatically more efficient: it may require more replicas, CPU, connections and model copies. A larger device can be wasteful at low load yet handle batching and bursts with fewer queueing steps.
| Placement goal | Preferred tendency | Cost to acknowledge |
|---|---|---|
| Consolidation | Pack replicas on already allocated nodes | Larger node-failure blast radius |
| Availability | Spread across nodes or zones | More residual capacity and traffic distance |
| Low latency | Reserve headroom for bursts | Lower steady-state utilization |
| Fast scale-down | Keep some nodes empty | Potential concentration risk |
Translate queue objectives into capacity headroom
A latency service objective is not satisfied by average utilization. As arrival rate approaches effective service rate, queues become sensitive to variation. Establish a maximum admitted utilization from replay or load testing for each replica shape, then size active capacity so forecast peak demand stays below it. Keep the autoscaler's observation and startup delays in the calculation. A replica that needs several minutes to load weights cannot rescue a burst that ends in thirty seconds.
Use a measurable rule: required replicas equals the ceiling of peak offered load divided by tested per-replica capacity at the target percentile, then add the stated failure allowance. If peak is 840 requests per second and a replica sustains 120 at the required latency, 840 divided by 120 equals 7 replicas. If one node must be survivable and the plan places two replicas per node, reserve at least two additional ready replicas or equivalent tested capacity. All figures are illustrative, and actual results require workload testing.
Worked example: capacity fits but the rollout does not
Assume, illustratively, four nodes with four identical GPU slots each. Twelve replicas use one slot, so raw occupancy is 12 of 16, or 75 percent. Packing four replicas on each of three nodes leaves one node empty and creates an easy scale-down target. Spreading three replicas on every node leaves one slot per node. Both layouts run twelve replicas, but the first loses four replicas when a full node fails while the second loses three.
Now consider a Kubernetes Deployment rolling update configured with maxSurge of four. If all other requests and placement constraints fit, the spread layout has four slots, one on each node. The packed layout also has four slots, all on the empty node, so maintenance or failure of that node can stall the surge. Failure of a serving node instead removes four packed replicas versus three spread replicas. Simulate steady state, one-node loss and the largest permitted rollout surge. Accept a layout only if each scenario meets minimum ready capacity and every new pod has a feasible node, not merely enough aggregate slots.
Use scheduler scoring without hiding hard constraints
Kubernetes NodeResourcesFit can score nodes with MostAllocated or a RequestedToCapacityRatio shape. Weighting an extended GPU resource more heavily can favor already occupied GPU nodes, which may free other nodes for scale-down. Scoring occurs after filtering. It cannot make an oversized replica fit, combine capacity across nodes or override a required topology-spread rule. Inspect effective scheduler configuration and test placement with the same sidecars and requests used in production.
Use hard topology spread only for availability requirements that justify Pending pods when a domain lacks capacity. Use ScheduleAnyway or preferred anti-affinity when imbalance is tolerable and admission matters more. Kubernetes notes that topology constraints can become imbalanced after pods are removed, and autoscaled-to-zero domains may be unknown to the scheduler. A descheduler or controlled replacement may be needed; scoring alone does not continuously rearrange running replicas.
Operate a small, explicit shape catalog
Keep a catalog with model version, precision, maximum context policy, batch policy, device profile, CPU and RAM requests, measured service capacity, startup time and failure-domain rule. Requalify a shape when any of those inputs changes. Track stranded slots by demand shape, Pending duration, queue age, tail latency, rollout headroom and replicas lost per node failure. These expose whether a denser layout is actually serving more traffic reliably.
A capacity marketplace can help compare measured shapes with available nodes, but procurement cannot resolve placement tradeoffs. A smaller catalog usually makes packing easier, while diverse models may still require incompatible profiles. Review the catalog against the actual arrival mix, not only each model in isolation. Track how often placements use the residual slots that the plan calls usable. Include rollout overlap and maintenance drains in capacity reviews, since both consume slots without increasing serving demand. Failure isolation deliberately leaves some capacity unused, and real traffic can invalidate a benchmark-derived utilization ceiling. Prefer a transparent policy that survives the documented failure scenario over the highest dashboard occupancy.