Throughput needs a qualification test
Raw throughput counts completed work, even when the work arrived too late or produced an unacceptable result. That is useful for studying engine saturation, but it is incomplete for capacity planning. Goodput adds a qualification: a request counts only if it finishes successfully, meets the defined quality rule, and stays inside the latency target. The denominator is wall-clock test duration, so the result remains a rate rather than a percentage.
Write the contract before running the load generator. For a streaming language model, the contract might require correct model and schema version, no server or client error, a task-specific quality pass, time to first token below 800 milliseconds, and time per output token below 50 milliseconds. These values are illustrative. Record end-to-end latency too, especially when output length is bounded. Changing any criterion creates a different goodput result.
Keep a request ledger with every disposition
Every offered request needs one terminal disposition: accepted, quality failure, latency failure, server failure, client timeout, cancellation, or unfinished at test end. A request that misses both quality and latency can carry both diagnostic flags while belonging to one primary outcome bucket. Do not drop timeouts from latency calculations or remove overload rejections from the offered count. Those omissions make a saturated service appear healthier as it sheds work.
MLCommons inference rules define a benchmark run around completed queries meeting latency and quality requirements, and its LoadGen server scenario produces arrivals at a target queries-per-second rate with a Poisson distribution. That is a useful reference for separating target load from valid performance, although a private service benchmark need not claim MLPerf compliance. Preserve arrival timestamps so the harness, gateway, and server clocks can be reconciled.
Report offered load, completed throughput, and goodput together
Offered load is all logical requests generated per second. Completed throughput is every terminal response per second, including responses that fail the acceptance contract if the server completed them. Goodput is accepted requests per second. Also report acceptance ratio, which is accepted requests divided by offered requests. A system can show stable completed throughput while goodput falls because queue delay pushes more requests past the limit.
Pair these rates with the failure mix and latency distribution. DistServe defines goodput around the maximum request rate that can satisfy service-level objectives and separates time to first token from time per output token. Its reported experimental gains are specific to its evaluated systems and should not be borrowed as expectations for another deployment. The durable lesson is the metric shape: useful capacity is load that remains inside stated objectives.
| Metric | Numerator | What it reveals |
|---|---|---|
| Offered load | All generated logical requests | Pressure applied to the service |
| Completed throughput | All terminal responses | Work the service finished, useful or not |
| Goodput | Quality-accepted responses meeting every latency target | Useful delivered capacity |
| Acceptance ratio | Accepted requests divided by offered requests | Share of demand converted into useful results |
Worked example: more completions, less useful work
In an illustrative 600-second run, a harness offers 72,000 requests, or 120 requests per second. Of the offered requests, 2,400 complete after the latency target, 600 complete but fail quality, 300 return errors, and 300 remain unfinished. Assume the categories are mutually exclusive for this summary. The service therefore completes 71,700 requests, and accepted requests are 72,000 - 2,400 - 600 - 300 - 300 = 68,400. Completed throughput is 71,700 / 600 = 119.5 requests per second, goodput is 68,400 / 600 = 114 requests per second, and acceptance is 68,400 / 72,000 = 95 percent.
At 130 offered requests per second for another 600 seconds, suppose 76,200 complete, exceeding the prior completed rate of 118 per second with 127 per second. Yet only 66,000 meet all criteria, so goodput falls to 110 requests per second and acceptance drops to 84.6 percent. The higher load is not a capacity win at the fixed target. All figures are illustrative rather than measurements, and production arrival shapes may produce a different knee.
Find the highest load that passes, not the busiest run
Warm the model and caches under a documented procedure, then run a staircase of offered loads using the same request-shape distribution. Randomize or repeat neighboring levels to reduce order effects. Each run must last long enough to include queue buildup and periodic system behavior. At every level report prompts, outputs, failures, unfinished work, latency percentiles, goodput, and achieved request rate. Keep the quality evaluator version fixed.
A simple decision rule is: rated capacity equals the highest offered load for which the acceptance ratio is at least 99 percent, each latency percentile meets its target, the quality score meets its floor, and two of two repeated runs pass. The 99 percent value and repeat count are illustrative. If 120 passes and 125 fails, test intermediate levels rather than quoting raw throughput from 125. Preserve a safety margin outside the benchmark result for failures and forecast error.
Make the result auditable and bounded
Publish the model and engine versions, hardware allocation, precision, parallelism, maximum lengths, request trace summary, arrival process, batching settings, timeout behavior, test duration, warmup, and every acceptance criterion. NVIDIA GenAI-Perf documents output token throughput, request throughput, time to first token, inter-token latency, and percentile summaries, showing why a generative benchmark needs more than one latency number. Its output is measurement evidence, not by itself a definition of product quality. Include the raw request ledger or a privacy-safe aggregate that reconciles every offered request. Without that reconciliation, reviewers cannot tell whether missing work was rejected, cancelled, timed out, or simply excluded from the report.
Goodput is still workload-specific. A fixed trace can go stale, a lenient evaluator can admit poor responses, and aggregate percentiles can hide one tenant or length band. Report segmented results for material request classes and rerun after model, prompt, scheduler, or maximum-length changes. A quality evaluator may itself be probabilistic or slow, so store its version and run it outside the timed serving path unless production actually performs the same check inline. When translating the passing frontier into GPU requirements, ClusterBid can help compare capacity options against that measured need. It cannot turn one trace and one latency target into a universal model rating.