Treat shutdown as a timed protocol
A serving worker should not interpret SIGTERM as an instruction to exit immediately. It is the start of a protocol with four distinct outcomes: stop accepting new work, let selected in-flight work finish, cancel work that no longer has a consumer, and release the process before the platform's hard deadline. Combining those outcomes in one signal handler produces races that appear only during rollouts or scale-downs.
Write the timing budget before writing the hook. If the pod has a 120 second grace period, an illustrative allocation might reserve 10 seconds for endpoint convergence, 90 seconds for draining, and 20 seconds for cancellation plus process cleanup. Those numbers are planning assumptions, not universal defaults. The sum must fit inside the platform deadline, and the last phase needs enough margin to flush telemetry and release shared memory.
Remove routing before draining
The worker's first state transition should be from ready to draining. Its readiness endpoint should then fail, while liveness remains healthy. Existing streams can continue, but admission code must reject or redirect every new generation. In Kubernetes, terminating endpoints are represented in EndpointSlices and ordinary traffic should stop selecting them, but load balancers, sidecars, and clients do not all observe that change at the same instant.
Do not use a blind sleep as the entire strategy. Record the time at which readiness changed, watch new-request arrivals, and start the drain window only after the arrival rate reaches zero for a short quiet interval. A practical decision rule is: proceed when no newly admitted request has appeared for two consecutive routing refresh intervals, or when the endpoint-removal allowance expires. The timeout prevents a faulty router from holding termination forever.
Classify in-flight work instead of waiting for everything
At drain start, snapshot each request's age, estimated remaining tokens or steps, deadline, client connection state, and priority. Continue work that has a connected consumer and a credible chance of finishing inside the drain budget. Cancel requests whose clients have disconnected, whose deadlines have passed, or whose projected completion exceeds the remaining time. This turns draining into a bounded scheduling decision rather than hopeful waiting.
For an illustrative worker with 90 seconds available, suppose request A has 20 seconds remaining, B has 70 seconds, and C has 140 seconds. With a 15 second cleanup reserve, the usable finish window is 75 seconds. A and B may continue if their clients are live. C should be rejected with a retryable response only if response headers have not been sent; an established stream must instead end according to the application's protocol and client retry contract, or be resumed elsewhere if the application supports it. Estimated remaining time is uncertain, so apply a safety factor derived from observed tail latency.
Propagate cancellation through the stack
Closing the HTTP connection is not sufficient proof that GPU computation stopped. Cancellation must travel from the frontend through the scheduler and backend to the generation loop. Each loop should check a cancellation token at a defined cadence, remove the sequence from future batches, and release its cache blocks. NVIDIA Triton documents request cancellation for long-running generative inference, but support depends on the client path and backend version in use.
Measure cancellation latency as a first-class service metric: time from client disconnect or drain decision to the request leaving the active batch. Page only when the p99 cancellation latency threatens the cleanup reserve; ticket slower regressions below that threshold. An honest limitation is that some kernels or backend calls are not safely interruptible mid-execution. In that case cancellation takes effect at the next boundary, so the grace period must cover the longest non-interruptible unit.
Align lifecycle hooks with signal handling
Kubernetes counts the preStop hook and normal process shutdown inside the same termination grace period. A hanging hook therefore consumes time the application expected to use for draining. Keep the hook small: call an authenticated local drain endpoint, verify that admission closed, and return. Let the application own the remaining state machine and handle repeated signals idempotently.
The process should treat a second termination signal as escalation, not as a second drain. Persist one shutdown reason and one start timestamp, stop background queue pulls, and keep health reporting explicit. If cleanup completes early, exit. If the drain deadline arrives, cancel remaining work and exit within the reserve. A watchdog should log the IDs and states of requests that survived each phase without logging prompts or generated content.
Test long generations and stuck clients
A useful shutdown test matrix includes a short request, a generation longer than the grace period, a client that stops reading without closing, a client that disconnects, a queued request not yet on GPU, and a backend call that ignores cancellation temporarily. Begin termination at different points in batch assembly and decoding. Verify both the client-visible result and the worker's final GPU memory, active-request count, and exit code.
Set measurable acceptance criteria. For example, after readiness becomes false, zero new requests may be admitted; 99 percent of eligible short requests must finish within the drain window; disconnected requests must leave active scheduling within 5 seconds; and the process must exit at least 10 seconds before forced kill. These are illustrative targets and should be replaced with values supported by the service's latency distribution and routing behavior.
Operate with a drain scorecard
For every rollout, report requests present at drain start, completions, cancellations by reason, forced terminations, new admissions after drain, and seconds of unused grace. A forced termination rate above zero is not automatically an incident, but repeated forced kills for requests that should fit the budget indicate a broken estimate, cancellation path, or router transition. Correlate the scorecard with deployment version and worker type.
Use a three-step diagnostic sequence when shutdowns fail. First, confirm the endpoint stopped receiving newly admitted work. Second, compare request remaining-time estimates with actual completion or cancellation latency. Third, inspect whether the process had cleanup time after draining. ClusterBid can help match replacement capacity to measured concurrency and startup requirements, but no capacity source fixes an application that keeps accepting work after it declares itself draining.