Forecasting GPU Spend from a Product Usage Funnel

A GPU forecast should begin at the event that causes model work. Registered accounts are usually too far upstream. Define a funnel such as monthly active users, active days per user, model-backed tasks per active day, and model calls per task. Keep each conversion as a named assumption with an owner and a source window. A product analyst can then change adoption without silently changing the infrastructure model.

Separate interactive calls from background work before doing arithmetic. A chat answer, document re-index, safety check, and evaluation run may share hardware, but they have different arrival patterns and deadlines. For each workload class, record completed tasks as well as attempted tasks. That distinction prevents an outage or retry storm from looking like healthy demand.

Token averages erase the expensive tail. Retain bands or a histogram for input tokens, requested output limit, actual output tokens, and any non-text dimensions such as image resolution. Prometheus documents that histograms retain counts in configured buckets plus a sum and count, while calculated quantiles are estimates. The practical lesson is to choose bucket boundaries around capacity-relevant thresholds and preserve raw samples for periodic model fitting.

Calculate demand by stratum. If 70 percent of calls are short and 30 percent are long, multiply the arrival rate of each group by its measured service demand. Do not apply one blended tokens-per-second number to both. Long prompts can change memory pressure, batching, and achievable concurrency, so a weighted sum of measured strata is more defensible than a single fleet headline.

Retries are compute demand even when the user sees one answer. Measure transport retries, application retries, speculative parallel attempts, and manual regenerations separately. A retry factor is attempts divided by accepted completions for the same workload class. Exclude requests rejected before GPU admission if they perform no model execution, and include canceled requests only to the extent that execution actually occurred.

Use an explicit diagnostic sequence when the factor rises. First compare attempted calls with completed calls. Then split failures by pre-admission, queued, executing, and post-response stages. Next compare client retry identifiers with server request identifiers. Finally inspect whether output length or latency shifted before the retry increase. This sequence distinguishes demand growth from duplicated work without assuming every error consumed a full request.

For each stratum, compute monthly attempted calls as active users multiplied by active days, tasks per day, calls per task, workload share, and retry factor. Divide that value by measured sustainable calls per GPU-hour under the required latency objective. NVIDIA Triton Performance Analyzer supports request-rate and concurrency load modes, which is useful because a service should be measured under the arrival pattern it must actually sustain.

Sustainable rate is not the highest point on a throughput curve. It is the rate that still meets the acceptance criteria for latency, errors, output quality, and memory stability. Record the model artifact, serving configuration, input data, concurrency or request-rate schedule, warm-up, and run duration. The resulting denominator is traceable and can be replaced without rewriting the product funnel.

Consider an illustrative planning scenario, not a price quote. Assume 40,000 monthly active users, 8 active days per month, 1.5 tasks per active day, and 1.2 model calls per task. That gives 40,000 x 8 x 1.5 x 1.2 = 576,000 accepted calls. Assume 75 percent short calls with a 1.05 attempt factor and 25 percent long calls with a 1.12 factor. Attempted volume is 576,000 x 0.75 x 1.05 = 453,600 short calls and 576,000 x 0.25 x 1.12 = 161,280 long calls.

Suppose repeatable workload tests show 900 sustainable short calls per GPU-hour and 180 sustainable long calls per GPU-hour at the service objective. Required execution is 453,600 / 900 + 161,280 / 180 = 504 + 896 = 1,400 GPU-hours. Add an illustrative 20 percent capacity allowance for maintenance, forecast error, and headroom: 1,400 x 1.20 = 1,680 budgeted GPU-hours. At a hypothetical blended charge of $2.50 per budgeted GPU-hour, planned compute spend would be 1,680 x $2.50 = $4,200. A real bill depends on whether capacity is metered as runtime, provisioned time, or a reservation. Storage, networking, and support may be billed separately.

Close each month by comparing forecast and observed attempted calls, stratum mix, retry factor, sustainable rate, and GPU-hours. Use a measurable rule: re-benchmark and reforecast when any workload stratum changes by more than 15 percent, its retry factor exceeds the modeled value by 0.05 for seven consecutive days, or observed GPU-hours differ from forecast by more than 10 percent for two monthly closes. The exact thresholds are policy choices, but they should be written before a miss occurs.

Capacity procurement needs a concurrency view as well as monthly hours. Convert the busiest representative window into required simultaneous GPUs, then apply the same service objective and headroom. When that measured requirement is ready for a supply decision, ClusterBid can help compare capacity that matches the workload rather than a generic GPU count.

This model assumes the measured workload remains representative. A new model, longer context window, changed batching policy, cache behavior, quantization, or hardware type can invalidate calls per GPU-hour. User growth may also correlate with a different task mix, so multiplying every stratum by one growth rate can understate the tail.

The forecast is therefore a controlled estimate, not a commitment that spend will equal one number. Publish a base case and named sensitivities, keep dollar rates explicitly hypothetical unless they come from an approved quote, and retain the query and benchmark artifacts. The useful output is not false precision. It is a chain from product behavior to compute demand that another person can inspect and update.

Histograms and summaries

Triton Performance Analyzer

Inference Load Modes