The average completion is not the average cost event
Two workloads can average 300 output tokens and impose very different capacity risk. One may cluster tightly around 300. Another may produce mostly 80-token answers plus a small set of 2,500-token reports. The second workload creates long-lived decode sequences, holds KV cache longer, and is more likely to overlap with other long requests. Pricing from mean tokens alone treats these tails as harmless even when they set the concurrency peak.
Build the forecast from joint request records rather than a single global average. At minimum, retain prompt tokens, generated tokens, end reason, model and version, service tier, queue time, time to first token, decode duration, retries, and whether the response passed the application acceptance check. vLLM's documented metrics include request-level histograms for prompt and generation tokens plus prefill, decode, queue, and end-to-end time, which illustrates the distinct dimensions a cost model needs.
Use buckets that preserve request shape
Create a two-dimensional matrix of prompt-length and output-length bands for each product route. A practical starting point might use output bands of 1 to 64, 65 to 256, 257 to 1,024, and more than 1,024 tokens, then split further wherever GPU-seconds per request change sharply. For each cell, store request share, p50 and p95 GPU-seconds, acceptance rate, and peak-hour arrival rate. Keep cancellations and failures as explicit outcomes.
NVIDIA GenAI-Perf supports synthetic input means and standard deviations, output-token means and standard deviations, and file-based prompts. Its archived 25.02 documentation also reports request throughput, output token throughput, time to first token, and inter-token latency. Synthetic distributions are useful for controlled sweeps, but a replayable production sample is preferable for pricing because output length depends on prompts, stopping behavior, and model responses. Preserve that relationship instead of sampling input and output lengths independently.
Translate each cell into useful GPU-seconds
For cell i, forecast raw GPU-seconds as request count multiplied by measured GPU-seconds per attempt. Then multiply by the observed attempts per accepted response, not merely attempts per HTTP success. Add a separately measured idle and failover reserve rather than burying it in an unexplained utilization factor. The total cost basis becomes the sum across cells plus reserve, orchestration overhead, and non-GPU costs that the business chooses to allocate.
Tokens remain a convenient billable unit, but they need not be the internal cost driver. Prefill and decode use the system differently, and two outputs with equal token count can take different time under different concurrency. Track contribution margin by cell: revenue units minus allocated GPU-seconds and other variable cost. If the long-output cell consumes 30 percent of peak GPU-seconds but earns 8 percent of revenue, a blended per-request price is sending the wrong signal.
Hypothetical scenario: the tail reverses the forecast
Assume an illustrative daily mix of 9,000 short requests and 1,000 long requests. Short requests generate 100 tokens and consume 2 GPU-seconds each; long requests generate 2,000 tokens and consume 30 GPU-seconds each. Total output is 900,000 + 2,000,000 = 2,900,000 tokens. Total compute is 18,000 + 30,000 = 48,000 GPU-seconds. The mean output is 290 tokens, but the 10 percent long segment consumes 62.5 percent of GPU time.
Suppose the internal planning assumption is a purely hypothetical $2.40 per GPU-hour and a 25 percent capacity reserve. Raw daily GPU cost is 48,000 / 3,600 x $2.40 = $32.00. Capacity-adjusted planning cost is $32.00 / 0.75 = $42.67. If 5 percent of long responses retry once, add 50 x 30 = 1,500 GPU-seconds before reserve. These figures are illustrative, not provider quotes; actual prices, efficiency, retry rates, and overhead vary.
| Segment | Requests | Output tokens per request | Total output tokens | GPU-seconds each | Daily GPU-seconds |
|---|---|---|---|---|---|
| Short | 9,000 | 100 | 900,000 | 2 | 18,000 |
| Long | 1,000 | 2,000 | 2,000,000 | 30 | 30,000 |
| Total | 10,000 | Not applicable | 2,900,000 | Mixed | 48,000 |
Choose billable units that customers can predict
Per-output-token pricing follows one visible source of work and naturally charges more for long completions. A split input-output token price can better reflect prefill and decode, provided the measurement is stable. Per-request pricing is easier to explain but needs length caps or bands. Reserved-throughput pricing can suit a customer with steady load, while charging for wall time exposes customers to queueing they do not control and is difficult to defend.
Choose a downside margin percentile that matches the business's risk tolerance and label it as a local policy, not a universal threshold. For example, a team might require the p10 contribution margin to remain nonnegative after retry work and reserved capacity in every material length cell. The customer's invoice should also be reproducible from request records. Review cells representing at least 95 percent of volume plus every cell representing more than 5 percent of GPU-seconds. If no simple unit passes, publish product tiers with explicit maximum output lengths rather than hiding cross-subsidy in one rate.
Reprice from observed drift, not isolated spikes
Run a monthly diagnostic in a fixed order. Compare the current output-length distribution with the pricing baseline. Recompute GPU-seconds per cell at representative offered load. Separate model-time changes from mix changes. Then calculate accepted-response cost and margin by cell, including retries and cancellations. Trigger review when the p95 output length or GPU-seconds per accepted response moves more than a chosen tolerance for two consecutive windows, rather than reacting to one unusual day.
This approach has limits. Historical distributions do not predict a new feature whose users request much longer answers, and a traffic replay can miss correlations during bursts. GPU-seconds also omit CPU, memory, network, storage, and support expense. Keep scenario bands around arrival rate and length mix, and rerun the serving benchmark after model, scheduler, quantization, or maximum-token changes. Compare weekday and peak-hour mixes as separate forecasts, and retain the original trace so finance can reproduce each revision. A quarterly average should not overwrite the short interval that determines how many concurrent workers must be funded. ClusterBid can then help match the resulting base and peak requirement to capacity without turning a hypothetical unit price into a market quote.