Freeze the decision question before running either version
A model upgrade is not cheaper because it produces more tokens per second in isolation. Define the unit of accepted product work first: a resolved ticket, an approved extraction, a correct code suggestion, or another outcome with a quality test. Then state the latency objective, traffic pattern, output policy, and cost boundary. Both versions must attempt the same unit under the same acceptance criteria.
Write a comparison manifest before testing. It should pin model artifacts, tokenizer and prompt template, serving software, precision, quantization, hardware, driver, runtime, request data, random settings where supported, and measurement window. If the upgrade requires a different template or output limit, record that as part of the treatment. The goal is not artificial sameness. It is to prevent an unrecorded configuration change from receiving credit for the model.
Replay one workload corpus with its original shape
Sample production-like requests with stable identifiers and retain the joint distribution of input length, requested output, task type, and arrival time. Do not independently shuffle columns because long prompts may correlate with certain tasks and output lengths. Remove or protect sensitive content according to the organization's data controls, and keep a frozen evaluation subset that neither version was tuned against during this comparison.
Run idle warm-up and steady-state windows consistently, then use the same request-rate schedule or concurrency points. NVIDIA Triton Performance Analyzer distinguishes concurrency mode, which maintains outstanding requests, from request-rate mode, which sends requests at a target rate. Choose the mode that represents the product, and keep it fixed. A closed-loop concurrency test can conceal queue growth that an open request-rate test would expose.
Put accepted quality in the denominator
Score each version with the same evaluator and human-review policy. Report both aggregate quality and slice results for critical tasks, languages, lengths, or safety categories. MLPerf Inference formalizes the principle that performance results live alongside accuracy requirements, including stricter high-accuracy variants for some benchmarks. An internal comparison does not need to copy MLPerf, but it should adopt the same discipline: speed outside the quality gate is not accepted throughput.
Define accepted outputs before seeing results. If one version completes 10,000 requests but only 9,200 pass while another completes 8,500 and 8,330 pass, raw throughput favors the first while accepted work is closer. Keep evaluator cost separate if it consumes additional compute, then include it in total serving economics if it is required in production.
Measure memory, concurrency, and latency together
For every load point, collect successful requests, failed requests, accepted outputs, input and output tokens, latency percentiles, queue time, GPU-hours, peak device memory, and host memory. Triton metrics distinguish request count, inference count, and execution count, which matters when batching combines requests. Record the exact semantic used in your calculation instead of treating these counters as interchangeable.
Memory can reverse an apparent win. A version that is faster at concurrency one may permit fewer replicas per GPU or reach its latency limit earlier because weights, activation workspace, or key-value cache use more memory. Sweep load until the latency or error objective fails, and repeat enough times to expose run-to-run variability. The comparison point is the highest sustainable accepted-work rate for each version, not necessarily the same configured concurrency.
| Measure | Why it stays in the scorecard | Failure signal |
|---|---|---|
| Accepted tasks per GPU-hour | Combines work, quality, and compute | Falls despite higher token rate |
| p95 and p99 latency | Protects user objective | Tail crosses target |
| Peak device memory | Constrains replicas and context | OOM or reduced concurrency |
| Failure and cancellation rate | Counts wasted work | Rises under load |
Worked example: calculate cost per accepted task
Consider an illustrative test with identical requests and a hypothetical GPU rate of $3.00 per hour. Version A sustains 720 completed tasks per GPU-hour and 96 percent pass the quality gate, yielding 720 x 0.96 = 691.2 accepted tasks per GPU-hour. Its compute cost per 1,000 accepted tasks is $3.00 / 691.2 x 1,000 = $4.34. Version B sustains 800 completed tasks per GPU-hour but 91 percent pass, yielding 800 x 0.91 = 728 accepted tasks per GPU-hour and $3.00 / 728 x 1,000 = $4.12.
Version B is about 5.1 percent cheaper on this illustrative compute measure: ($4.34 - $4.12) / $4.34. Actual prices vary, and this excludes storage, networking, engineering, and required evaluator costs. Suppose B also needs 25 percent more device memory and can run only one replica where A runs two. The single-process rate does not settle fleet economics. Re-run at the deployable replica count and required context mix before declaring B cheaper.
Apply a two-gate decision rule before rollout
Use a quality gate first and an economics gate second. An illustrative rule is: no critical evaluation slice may regress by more than 1 percentage point, overall accepted-task rate must improve or remain neutral, p99 latency and error rate must meet the existing objective at forecast peak load, and peak memory must fit the deployment envelope with 10 percent headroom. Only after those conditions pass should cost per 1,000 accepted tasks determine the winner.
If both versions pass, require a material advantage larger than measurement noise. For example, approve the upgrade for efficiency only when the upper bound of its repeated-run cost estimate is at least 5 percent below the lower bound for the current version. Otherwise classify the result as inconclusive or approve it for a separately documented quality benefit. ClusterBid fits after this measurement step, when the team can match capacity options to an evidenced hardware and concurrency envelope.
Diagnose regressions in a fixed order
When results disappoint, first verify artifact, tokenizer, prompt template, precision, and serving configuration. Second compare output length and stop behavior, because longer answers can look like a compute regression. Third compare quality slices. Fourth inspect queue time, batch execution, failures, and cancellations across load points. Fifth inspect peak memory and replica count. Only then tune version-specific settings, with every tuning change logged and both versions rerun when fairness requires it.
No benchmark can guarantee production behavior. Traffic correlations, cache hits, tenant interference, thermal conditions, and downstream timeouts may differ from the replay. Evaluators can also miss important quality changes. Use a limited rollout with the same scorecard, rollback trigger, and version attribution. Recheck after a meaningful traffic-mix change instead of treating one favorable benchmark as permanent evidence.