Benchmarking LLM Serving with Real Prompt Length Distributions

A test in which every request has 1,024 input tokens and produces 128 output tokens is repeatable, but it represents a single point, not a service. Production traffic usually mixes short chats, long retrieved contexts, brief classifications, and long generations. Prefill cost responds strongly to input length, while decode occupancy responds to generated length. Averages conceal the long requests that hold cache and scheduler capacity.

Preserve the joint distribution rather than sampling input and output lengths independently. Long prompts may correlate with short answers, or long prompts may also request long answers. Breaking that relationship invents traffic. Build records containing tokenized input length, actual output length, arrival offset, request class, cache eligibility, and relevant generation settings. Remove payload text if privacy demands it, but retain the structural fields required for replay.

Token length alone does not preserve every cost driver. Sampling options, stop sequences, number of returned candidates, adapters, and image inputs can alter work or memory even when text lengths match. Add fields only when the service actually uses them, then define a fallback for records the candidate cannot replay. Excluding them silently changes the denominator and can make a new stack look faster.

Sample completed production requests across ordinary and peak periods. Tokenize with the exact tokenizer used by the candidate deployment. Store lengths after the actual chat template and retrieval assembly, because character counts and pre-template token counts are weak substitutes. Stratify rare but important classes so the corpus includes them, then attach weights if the sample is intentionally disproportionate.

A useful manifest records sampling window, tokenizer hash, model-facing template version, filters, truncation policy, and counts by length band. Keep failure and cancellation semantics explicit. Actual output length for a canceled stream is not the length it would have generated, so decide whether the benchmark models user abandonment or unconstrained generation. Do not silently replace every output with max_tokens; that measures a stress boundary, not observed demand.

Validate the corpus before running it. Compare its joint input and output length histogram with the source window, plus request-class shares and interarrival percentiles. Set a measurable acceptance rule, such as no important class differing by more than two percentage points and p95 lengths remaining within five percent. Those tolerances are illustrative. Publish deviations so readers can judge whether a result applies to their service.

Closed-loop tests that maintain N concurrent requests immediately replace each completion. They are useful for saturation studies but do not reproduce an open-loop service where arrivals continue during slowdowns. The vLLM 0.8.5 serving benchmark exposes request-rate and burstiness controls and reports TTFT, time per output token, inter-token latency, and end-to-end latency. Its synthetic arrival process is useful for controlled load, while a timestamp replay can preserve an observed arrival trace more directly.

Use at least three scenarios: observed median load, observed peak-window arrivals, and a controlled overload sweep. Keep corpus order when it encodes bursts or request-class clustering. For an illustrative trace with arrival offsets 0, 40, 55, and 310 ms, send at those offsets rather than spacing four requests evenly across 310 ms. The difference can expose queue formation even though the average arrival rate is similar.

Preserve offered arrivals when the server slows. If the client waits for a response before scheduling the next request, it accidentally converts the test into a closed loop and hides queue growth. Instrument client-side send time, connection time, first byte, first token, and completion. A saturated load generator can also become the bottleneck, so verify its CPU, socket, and event-loop headroom separately from the server.

Load the engine, execute representative shapes until graph capture, kernel selection, and memory pools settle, then begin measurement. Report the warmup procedure separately. A single short warmup request is inadequate when the serving stack captures several batch or sequence shapes. Conversely, including model download and first initialization in steady-state latency confuses deployment readiness with serving performance.

Choose cache semantics deliberately. A cold-prefix run measures no reuse; a production-order replay may reproduce repeated prefixes; a shuffled replay can destroy locality. Report which one you used. Resetting the server between every request is usually unrealistic, while carrying state across candidate configurations can be unfair. Fixed seeds help reproducibility, but deterministic token IDs may not reproduce application-level stop conditions or tokenizer work.

Report offered requests divided by the scheduled arrival interval, completed requests divided by the measured interval, input tokens processed per measured second, generated tokens per measured second, and errors over all offered requests. Also report TTFT, time per output token, inter-token latency, and end-to-end latency, with p50, p95, p99, and sample counts. Then condition results on input-length band, output-length band, and request class. A global p95 can improve merely because traffic composition shifted toward short prompts.

Goodput counts requests meeting declared service objectives. An illustrative rule might require TTFT at or below 800 ms and time per output token at or below 60 ms. If 9,400 of 10,000 completed requests meet both during 600 seconds, goodput is 9,400 / 600 = 15.67 requests per second, while compliance is 94 percent. These figures are illustrative and are not a measured product result. Publish failures and timeouts in the denominator policy.

Percentiles need uncertainty context. Report the number of observations in each band and repeat the run, especially for sparse long-prompt classes. Ten observations cannot support a stable p99 claim because the result is effectively one extreme sample. For small classes, show raw counts, maximum, and confidence intervals or defer the percentile until the corpus contains enough requests.

Increase offered load in steps while holding the replay corpus and configuration constant. Select the highest load whose required percentiles, error rate, and memory behavior remain inside policy, then apply an explicit resilience margin. A measurable stop rule is two successive load steps with p95 TTFT above the objective or any sustained OOM. Repeat trials and show dispersion rather than selecting the best run.

Use the same test setup for candidate hardware, but do not assume one corpus remains representative forever. Prompt templates, retrieval limits, user behavior, and model stop behavior can shift lengths. Refresh the corpus on a schedule and after major product changes. Replay also has an honest limitation: retained lengths and arrivals do not reproduce every content-dependent kernel path, network condition, cache key, or cancellation decision. Pair replay with production canaries before a broad change, and provide any capacity supplier with the corpus summary, test configuration, and goodput threshold rather than only a model name.

vLLM 0.8.5 benchmark_serving.py

vLLM 0.8.5 Production Metrics

BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems

vLLM vs SGLang vs TensorRT-LLM 2026 Inference Showdown: Throughput, Latency, Cost

A10 24GB vs A100 80GB GPU Comparison 2026: Entry Ampere vs Flagship - Performance, Price and Best Workloads