Cost per Successful Request Beats Cost per Generated Token

Cost per generated token answers a narrow utilization question: how much spend was allocated to emitted tokens. It does not say whether the caller received a valid answer before its deadline. A timed-out generation can emit tokens after the user leaves. An invalid structured response may trigger a full retry. A canceled stream may consume prefill and decode work but contribute no accepted outcome. Comparing providers or stacks on generated tokens alone rewards failure that produces tokens efficiently.

Define success at the application boundary. For a structured extraction service, success might require valid schema, required fields, task-level checks, and completion under a latency objective. For chat, it might require a completed or deliberately stopped response delivered before the client deadline. Keep model-quality evaluation separate from transport success, but include both when they determine whether the request must be repeated or discarded.

Start with unique logical requests, not HTTP attempts. Link every retry to its original request through an idempotency or trace identifier. For each logical request record attempts, prompt and generated tokens, terminal status, validation result, client cancellation, latency objective result, and whether a later attempt superseded an earlier one. The denominator is the count meeting the declared success predicate exactly once.

Do not classify every cancellation as infrastructure failure. A user may stop because enough information arrived, or because the interface changed context. Preserve cancellation reason and whether the result was still accepted. Likewise, a safety refusal can be a correct product outcome or a failure, depending on the contract. Publish the predicate beside the metric so teams cannot improve the number by silently weakening the meaning of success.

The numerator should include accelerator time, allocated idle capacity, CPU and memory for the serving path, storage and network attributable to requests, and operational overhead that changes between alternatives. At minimum, include every attempt, including failed retries and work performed after a disconnect. If the comparison is limited to variable inference cost, label it that way and do not present it as total cost.

Allocate shared capacity with a stated method. GPU-seconds by model replica is often more defensible than dividing a monthly bill by tokens, because reserved replicas can be idle. Track unsuccessful request categories alongside cost. NVIDIA Triton documents success and failure counters, pending count, request duration, and queue duration, with failure reasons including rejected, canceled, backend, and other. Application validation still has to be added outside the server because runtime success does not prove useful output.

Consider 100,000 logical requests. Option A has an illustrative compute allocation of $900, plus $100 of other attributable serving cost, for $1,000 total. It generates 50 million tokens, so its generated-token cost is $1,000 divided by 50,000,000, or $0.000020 per token. Of the logical requests, 94,000 complete, 3,000 fail validation, 2,000 miss the latency objective, and 1,000 are canceled without an accepted result. Cost per successful request is $1,000 divided by 94,000, or about $0.01064.

Option B uses an illustrative $1,050 compute allocation and $100 of other cost, totaling $1,150. It generates 52 million tokens, giving about $0.00002212 per token, which looks worse than A. But suppose 98,000 requests succeed, 1,000 fail validation, 600 miss latency, and 400 are unaccepted cancellations. Its cost per successful request is $1,150 divided by 98,000, or about $0.01173. B still costs more per success, but about 10.3 percent more, compared with about 10.6 percent more under the token metric, and it delivers 4,000 more outcomes. All dollar figures and rates are hypothetical planning assumptions; actual prices and workloads vary.

Add an attempt amplification ratio: total attempts divided by logical requests. If 100,000 requests produce 108,000 attempts, the ratio is 1.08. Then measure wasted accelerator-seconds for attempts whose outputs were neither accepted nor needed. A retry may be cheap in output tokens yet expensive in repeated prefill, especially with long prompts. Generated-token accounting misses that input-side work unless the billing model explicitly includes it.

A decision rule can require a candidate to reduce cost per successful request by at least 5 percent without worsening the success rate or p95 latency beyond agreed tolerances. Use confidence intervals or repeated windows when volumes are low. If a change reduces cost per token but increases attempt amplification by more than 2 percentage points, hold deployment and inspect validation failures, timeout thresholds, and disconnect handling before attributing the result to model efficiency.

First verify the success predicate and request linkage did not change. Second split the numerator into allocated capacity and per-attempt work. Third split unsuccessful outcomes into admission rejection, runtime error, invalid output, deadline miss, and cancellation. Fourth compare prompt and output distributions within each category. Fifth inspect queue time versus compute time. This order prevents a traffic-mix shift or instrumentation change from masquerading as a GPU regression.

The metric has an honest limitation: it compresses unequal outcomes into a binary denominator. One accepted answer may be far more valuable or difficult than another. Segment by task, tenant tier, model, and length bucket before comparing. Also retain cost per token and hardware utilization as diagnostic metrics. Cost per successful request is the decision metric for delivery efficiency, while the lower-level metrics explain why it moved and help ClusterBid match capacity to the workload actually succeeding.

Metrics | NVIDIA Triton Inference Server

Statistics Extension | NVIDIA Triton Inference Server

Production Metrics | vLLM 0.6.1