A closed socket does not cancel GPU work by itself
A user can close a browser, a mobile radio can disappear, or a client deadline can expire while an inference engine continues decoding. The transport knows the consumer is gone, but the scheduler owns the sequence and its KV-cache blocks. Unless cancellation crosses every boundary, generated tokens can be discarded at the gateway while the engine still batches the abandoned sequence with live work. Streaming makes this gap visible because the request remains active for many scheduling steps.
Model cancellation as a control message with an identity, not as an implied side effect of a failed write. The same request ID must connect client, edge proxy, application gateway, serving frontend, and engine. Each layer should record when it first observed abandonment and when it forwarded cancellation. This provides a latency budget for reclamation and exposes the boundary where the signal disappeared.
Trace cancellation through every hop
The client may cancel an HTTP request or simply stop reading. A proxy may notice immediately, after a buffered write, or only when its own timeout fires. The application must observe the disconnect, stop producing response chunks, and invoke the engine's abort operation. The engine then removes or marks the request so the scheduler no longer selects it. Finally, request-owned cache blocks and frontend state must become reclaimable.
Instrument six timestamps: request accepted, first token emitted, client disconnect observed, gateway cancellation sent, engine abort acknowledged, and request state released. Do not infer success from an HTTP 499 alone. That status can describe the frontend outcome without proving that decode stopped. The proof is that the request ID disappears from running and waiting sets and its generated-token counter stops increasing within a bounded number of scheduler iterations.
| Boundary | Expected event | Evidence |
|---|---|---|
| Client to proxy | Stream closes or deadline fires | Client trace and proxy disconnect log |
| Proxy to frontend | Cancellation reaches handler | Shared request ID and timestamp |
| Frontend to engine | Abort is invoked | Abort counter and request ID |
| Engine to allocator | Sequence leaves schedule and state frees | Running count, token stop, cache recovery |
Count work after abandonment
The primary waste metric is output tokens computed after the earliest trustworthy disconnect signal. Count tokens at the engine, because bytes written by the gateway exclude discarded output. Also track post-disconnect scheduler steps, milliseconds until abort acknowledgement, and milliseconds until memory release. Separate requests cancelled before first token from those cancelled during decode; long prefill can dominate the former, while repeated decode steps dominate the latter.
Use two views. Wasted-token ratio equals post-disconnect generated tokens divided by all generated tokens in the same interval. For cancelled requests, also report total engine-generated tokens beside total tokens delivered to the chosen observation point. If a ratio of those totals is published, define it only when delivered tokens are greater than zero and report zero-delivery cases separately. Network buffering means a token accepted by a proxy was not necessarily consumed by the user, so avoid presenting delivered bytes as a perfect measure of user value.
Worked example: a slow cancellation path
Consider an illustrative stream decoding at 40 tokens per second when the client disconnects. The proxy detects the closure after 250 milliseconds, the gateway takes 150 milliseconds to notify the engine, and the scheduler applies abort 100 milliseconds later. Total cancellation latency is 0.25 plus 0.15 plus 0.10, or 0.50 seconds. If the rate stays constant during that interval, the estimate is 40 multiplied by 0.50, or 20 wasted tokens for that request. It is not an upper bound unless 40 tokens per second is itself a proven upper bound.
If 30 such cancellations occur each minute, the illustrative waste is 600 tokens per minute. Actual batching and scheduler timing make this estimate approximate. Set a decision rule around directly measured data: alert when p95 disconnect-to-abort exceeds two scheduler cycles or when the wasted-token ratio exceeds 1 percent for three consecutive five-minute windows. Those thresholds are examples; choose limits based on service cost, normal client behavior, and telemetry precision.
Test deliberate cancellation, not generic errors
Build a test client that starts a stream, reads a chosen number of tokens, then closes the connection without waiting for completion. Cover cancellation during queueing, prompt processing, early decode, and late decode. Repeat with client deadlines, proxy idle timeouts, and explicit user cancellation because they may follow different code paths. Preserve request IDs and use a low enough rate that each abandoned request can be traced without ambiguity.
The pass condition should cover behavior, not just status codes. Within the declared reclamation interval, no additional output tokens should accrue, the request should leave engine queues, active sequence count should fall, and reusable cache capacity should recover. Healthy parallel streams must continue without a latency spike beyond the agreed tolerance. Include a control request that completes normally so the test distinguishes cancellation cleanup from a general engine failure.
Verify prompt and cache cleanup separately
Cancellation during a long prompt can leave different artifacts from cancellation during decode. The frontend may retain prompt text or token IDs, while the engine may hold partially computed KV state. Inspect process memory, engine request maps, scheduler queues, and cache-block availability after repeated cancellations. Memory need not return to the operating system if an allocator keeps a pool, so a flat process-resident-memory graph is not sufficient evidence of a leak.
Run an illustrative soak of 1,000 sequential cancellations at a fixed prompt size, followed by a normal request set. Compare the number of live request objects and available cache blocks before and after, allowing for documented caches and allocator high-water behavior. Fail the test if live per-request state grows monotonically or normal requests lose capacity after the workload stops. Do not require device memory to become zero if the engine intentionally retains its KV pool.
Rely on an engine contract, then test the integration
The versioned vLLM AsyncLLMEngine documentation exposes an abort method and shows checking whether a request is disconnected before aborting its request ID. That establishes an engine mechanism, not an end-to-end guarantee. Middleware, buffering, proxies, and custom streaming wrappers can change disconnect propagation. Treat the exact serving stack as the unit under test and rerun the suite whenever a gateway, ASGI layer, or engine version changes.
Cancellation cannot reclaim a kernel already executing in the middle of a scheduler step. The practical goal is bounded stop latency at a safe scheduling boundary, not instantaneous preemption. Some completed prefix blocks may also remain cached for reuse after request state is released; that is not necessarily leakage. Document which memory is request-owned, which is pooled, and which is intentionally cached so alerts measure failed reclamation rather than normal engine design.