Write the capacity decision before the test
A concurrency test is useful only if it resolves a specific production choice. State the current limit, proposed limit, affected model, serving configuration, and pass criteria before generating traffic. Concurrency is not a workload by itself. Ten active requests with short prompts and capped outputs can be easier than three requests carrying large contexts. The test must represent the token demand and arrival behavior that the limit is intended to admit.
Create a one-page contract: increase maximum active requests from X to Y only if the candidate sustains the target arrival rate, p95 time to first token, p95 inter-token latency, error rate, queue bound, and memory margin for the full steady interval. Include an automatic stop condition for errors or runaway queueing. Without this contract, teams tend to select the nicest graph after the run.
Define the request population in tokens
Build a request matrix from production distributions, not a single average prompt. Record prompt-token percentiles, requested and actual output lengths, streaming share, sampling settings, model or adapter mix, cached-prefix share, and cancellation rate. Preserve correlations. Long prompts may also request long outputs, and flattening those two distributions independently can create a workload that never occurs. Keep a named corpus and pin the tokenizer and chat template used to count it.
Protect sensitive data by using structurally representative synthetic or approved replay inputs. Validate the corpus at the serving boundary because templates add tokens that raw user text does not show. A practical minimum is separate short, typical, long, and extreme bands with explicit proportions. Report achieved prompt tokens per second and output tokens per second alongside requests per second so another team can reproduce the demand.
| Workload field | Why it changes capacity | Required record |
|---|---|---|
| Prompt length | Changes prefill work and KV allocation | Token histogram after templating |
| Output length | Changes decode duration | Requested and completed token histograms |
| Arrival process | Controls overlap and queue pressure | Timestamped schedule and achieved rate |
| Prefix reuse | Can reduce repeated prefill | Hit ratio and exact shared-token count |
| Cancellation | Changes useful and wasted work | Cancellation phase and rate |
Choose an arrival process that cannot hide saturation
In a closed-loop test, each virtual user waits for its request to complete before starting another. When the server slows, the generator sends fewer new requests. This feedback can make an overloaded service appear stable because offered load falls with response speed. Grafana k6 documents this as a drawback of the closed model and implements open-model arrival-rate executors that start work independently of prior response duration.
Use an open arrival process when production users or upstream jobs arrive independently. Include realistic bursts rather than perfect spacing if bursts matter operationally. A closed model remains useful for a fixed population that truly waits before submitting again, but label it clearly. For concurrency admission, report offered arrivals, started iterations, dropped iterations, completions, and queue depth. Achieved throughput alone omits demand the generator failed to start.
Separate warmup, steady state, and recovery
Warmup can include model loading, kernel compilation, allocator growth, cache population, and autoscaler reactions. Decide which effects users normally encounter. If production replicas join cold, test cold-start readiness separately. For a steady-state capacity run, warm until throughput, cache occupancy, and latency stop showing a directional trend, then exclude that interval from the primary statistics while retaining it in the record.
Run the measured phase long enough to observe cache turnover, periodic telemetry, background maintenance, and thermal or power behavior. Five minutes may expose immediate saturation but miss slow queue growth. Add a recovery phase at low traffic and require queue depth, active sequences, and latency to return near baseline. A test that ends at peak load cannot show whether the service sheds backlog or remains degraded.
Prove the client is not the bottleneck
The load generator must tokenize inputs if required, hold streaming connections, parse chunks, record timings, and schedule arrivals without falling behind. Monitor its CPU, memory, network, open files, event-loop delay, and dropped-iteration counter. Synchronize clocks across distributed generators and the service. If the generator reaches a resource ceiling, a flat request rate says nothing about server capacity.
Apply a measurable headroom rule before accepting results: generator CPU below 70 percent, no resource exhaustion, scheduled arrivals within 2 percent of target, and dropped iterations equal to zero during the measured interval. These values are illustrative. Validate them for the chosen tool. Repeat one stage with twice the generator capacity; if server-side results change materially at the same offered load, the earlier client was not trustworthy.
Worked example: test a move from 32 to 48
Assume an illustrative service is capped at 32 active requests and the proposal is 48. Production arrivals peak at 6 requests per second. The test uses 60 percent 500-token prompts, 30 percent 2,000-token prompts, and 10 percent 8,000-token prompts, with an illustrative mean of 120 generated tokens. Stages run at 4, 5, 6, and 7 arrivals per second after a 15-minute warmup, each for 30 minutes, followed by 10 minutes of recovery.
At a steady 6 completed requests per second, mean completed output volume is 6 multiplied by 120, or 720 output tokens per second. Six arrivals per second alone does not guarantee that completion rate under saturation, so report arrivals, completions, and generated tokens separately. Pass 48 only if p95 time to first token stays under the declared service limit, p95 inter-token latency stays within 10 percent of the 32-request baseline, errors remain below the agreed threshold, queue depth has no positive trend, and generator headroom passes. All values are illustrative, and actual capacity depends on model, hardware, software, and traffic.
Stop on failure and publish the uncertainty
Declare abort conditions for out-of-memory events, sustained queue growth, severe latency, excessive errors, or unhealthy replicas. Record server configuration, engine version, model revision, GPU type, replica count, batching limits, cache settings, and raw arrival schedule. Do not silently retry failed requests during the main run because retries alter offered load. If production clients retry, model that behavior in a separate scenario with bounded policy and explicit accounting.
A passing load test does not prove capacity for every prompt or future release. Synthetic text may tokenize differently, prefix-cache hits may drift, and a test cluster may lack noisy neighbors or network constraints. Treat the result as a dated operating envelope for a pinned configuration. Raise concurrency gradually, keep rollback simple, and compare production distributions with the tested corpus before assuming the envelope still applies.