A running process is not necessarily a serving replica
A GPU serving container can open its HTTP port while model weights are still downloading, deserializing, compiling, moving to device memory, or warming kernels. A TCP probe or process check may therefore mark the pod ready before it can answer the route's real model request. The result is avoidable 404, unavailable, timeout, or first-request latency during rollout. Pod readiness should represent the capability the load balancer is about to use.
Keep three states separate. Liveness asks whether the process is making enough progress to remain running. Artifact readiness asks whether the required model version and dependent files are present and verified. Serving readiness asks whether the replica can accept the supported request shape within a defined bound. Expose the reason for each false state in metrics or logs without placing secrets in the probe response. Combining them into one vague health endpoint makes slow startup look like a crash and makes a listening socket look like a usable model.
Give startup, liveness, and readiness different jobs
Kubernetes documents that a startup probe disables liveness and readiness probes until it succeeds. Use it to protect a legitimately slow initialization path from premature restarts. Its failure window should cover a measured high percentile of valid model startup plus operational margin, not an arbitrary long delay. Once startup succeeds, liveness should detect a stuck process with a cheap check that does not depend on every optional downstream service.
Readiness runs throughout the container lifecycle and removes an unready pod from matching Service endpoints. It should fail while required artifacts are absent, the selected model version is unavailable, device initialization is incomplete, admission is paused, or the replica is draining. Avoid a heavyweight inference on every probe interval. A cached internal state plus a periodic representative self-test can provide model-aware evidence without creating a probe-driven workload. Give the self-test a freshness limit so an old success cannot keep a newly stuck backend in rotation indefinitely.
Name the model, version, and usable state
A multi-model server needs a readiness contract more precise than server ready. Define the required set per route or deployment. For each required model, check expected name and version, artifact manifest or digest, successful backend load, required device placement, and absence of a terminal load error. If optional models are missing, expose that separately rather than making an unrelated route unavailable.
NVIDIA Triton exposes server and model readiness through its inference protocol and supports model control modes that determine when models load. Its documentation states that models failing to load can be marked unavailable, and strict readiness can make the server readiness endpoint depend on selected models being loaded. Use those semantics deliberately. A custom gateway may still need route-level readiness when one pod hosts models with different availability requirements. Keep model repository mutation behind trusted controls, since readiness is an observation and not an authorization boundary.
Worked example: size the startup window from observations
Assume an illustrative deployment has 40 measured cold starts for one model. The slowest valid artifact download and load completes in 430 seconds, while corrupted artifacts fail within 70 seconds. Choose a startup probe with periodSeconds set to 10 and failureThreshold set to 48, giving a nominal failure budget of 10 x 48 = 480 seconds after any initial delay. Exact kill timing also depends on probe execution and timeout. These numbers are illustrative operational assumptions, not product performance claims. Actual startup depends on artifact size, storage, compilation, GPU, and network.
After startup, readiness uses periodSeconds of 5, successThreshold of 2, and failureThreshold of 1. Kubernetes may run readiness probes more frequently than periodSeconds while a container is not ready, so five seconds is not a guaranteed transition delay. A cached representative self-test runs internally every 30 seconds. The decision rule is: ready only when the required digest is loaded, the model-specific ready state is true, the last self-test is newer than 60 seconds and successful, and the pod is not draining. Liveness remains true during a transient artifact-service outage after the model is loaded, preventing an unnecessary restart.
| Signal | Question answered | Failure action |
|---|---|---|
| Startup | Has valid initialization completed within its budget? | Restart after the measured startup window expires |
| Liveness | Is the serving process making progress? | Restart after repeated process-health failure |
| Artifact state | Is the required model version loaded and verified? | Keep route unready; report load cause |
| Readiness | Can this replica accept new requests now? | Remove from regular traffic without killing it |
Drain requests before unloading the model or exiting
On rollout or scale-down, set the application-level draining flag first so readiness becomes false and new work stops. Continue serving admitted requests, track the in-flight count, and exit only when it reaches zero or the explicit drain deadline arrives. Kubernetes termination flow marks terminating endpoints not ready for regular traffic and runs a preStop hook before termination when configured, but endpoint propagation and external load balancers are not instantaneous.
Set terminationGracePeriodSeconds to cover readiness propagation, the longest admitted request under policy, and shutdown overhead. Remember that the grace-period countdown includes preStop execution. Reject or cap new streaming and long-running requests before the drain begins if they cannot finish within that budget. Record forced cancellations separately. A sleep-only preStop hook guesses at propagation; an application-aware drain endpoint supplies evidence about whether work has actually completed.
Test transitions, not just the green state
Exercise cold start with a normal artifact, a missing object, a wrong digest, a slow download, an out-of-memory load, and a backend initialization error. Verify that startup and readiness change as designed and that liveness does not create a restart loop for recoverable conditions. During rollout, send controlled requests while a pod enters drain and confirm that no new requests are admitted after endpoint convergence and that admitted work completes within the grace period.
Readiness cannot prove every request shape will succeed or that an external load balancer instantly respects endpoint changes. Model load success also does not prove acceptable latency under concurrency. Pair readiness with load tests, request metrics, and rollout capacity checks. Alert separately on pods that stay live but unready beyond the startup or recovery objective, because silent capacity loss can otherwise accumulate. The useful contract is narrower: traffic is offered only to replicas with the required model available, and termination follows a measurable drain sequence rather than process status alone.