Time to First Token Budgets for Retrieval-Augmented Generation

Time to first token begins when the client sends a request and ends when that client receives the first streamed token. A server-side model timer is narrower. It excludes request transit, authentication, retrieval, reranking, prompt construction, scheduler waiting, and response transit. A fast prefill can therefore coexist with a slow application. Define the service-level indicator at the client, with a monotonic clock, before assigning component budgets.

Use percentiles rather than averages. Report p50 to describe the ordinary path and p95 or p99 to expose queues and slow dependencies. Slice by region, prompt-token band, retrieval mode, cache outcome, and model deployment. Never add independently calculated p95 component values and call the result end-to-end p95. Percentiles are not additive because the slow requests in each stage may be different requests. Join spans by request ID and calculate the complete distribution from individual traces.

Treat the target as an envelope with named owners. For an illustrative 1,200 ms end-to-end p95 TTFT target, a planning ledger might reserve 120 ms for inbound and outbound transit, 180 ms for embedding and vector retrieval, 140 ms for reranking, 80 ms for prompt assembly and policy checks, 220 ms for admission and model queueing, 400 ms for prefill and the first decode step, and 60 ms of contingency. The arithmetic is 120 + 180 + 140 + 80 + 220 + 400 + 60 = 1,200 ms. These stage allowances are planning assumptions, not component p95 measurements and not measured performance.

A budget is useful only if it has a boundary, a metric, and an action. If retrieval consumes 260 ms, knowing that the model used 330 ms does not identify the retrieval cause. Split retrieval into query embedding, datastore transit, search execution, document fetch, and serialization when that subsystem misses its allowance. Keep contingency visible rather than quietly spending it; repeated contingency use signals that a formal budget is too small or an unmeasured step exists.

Record timestamps at the client send, edge receive, retrieval start and finish, rerank start and finish, inference enqueue, inference scheduled, prefill finish, first token emitted, edge flush, and client receive. Propagate one trace identifier across services and attach input-token count, retrieved-document count, reranker candidate count, cache status, worker ID, and response region. Synchronize hosts, but calculate durations on one host with a monotonic clock wherever possible to avoid wall-clock skew.

Serving metrics provide a useful cross-check. vLLM 0.8.5 documents histograms for request queue time, prefill time, inference time, and prompt tokens. Those metrics describe the engine, while application spans describe everything around it. Compare the client TTFT with the sum of per-request traced durations. A persistent residual above the 60 ms illustrative contingency points to missing instrumentation, buffering, clock mismatch, or serialization rather than automatically to GPU work.

Retrieval can be quick yet make TTFT slow by supplying too many tokens to prefill. Track both retrieval duration and the number of tokens admitted after reranking. A fixed top-k rule hides this coupling because document lengths vary. Prefer a token budget and record discarded candidates. If the prompt template contributes 600 tokens, the user query 200, and the application permits 3,200 input tokens, the illustrative evidence allowance is 3,200 - 600 - 200 = 2,400 tokens.

Build a curve by prompt-token band under a controlled arrival rate. If p95 engine TTFT is 310 ms for 0 to 2,000 prompt tokens, 470 ms for 2,001 to 4,000, and 820 ms above 4,000, the diagnostic response is to inspect evidence size before buying capacity. These example values are illustrative. They do not claim a particular model or GPU result. Recheck answer quality when trimming context because lower latency that removes decisive evidence is not a successful optimization.

First confirm the client-side regression and its traffic slice. Second compare network spans. Third inspect retrieval and reranking independently. Fourth compare prompt-token distributions. Fifth inspect inference queue time, then prefill time. Sixth check first-token buffering at proxies and SDKs. Change one control at a time. This order prevents a queueing incident from being mislabeled as a retrieval incident merely because both appear before the first token.

Use a measurable rule: when p95 TTFT exceeds 1,200 ms for three consecutive five-minute windows and at least 200 requests, identify the largest per-request budget overrun by excess milliseconds. If queue p95 is over its 220 ms illustrative allowance while prefill stays within 400 ms, test admission control or capacity. If prefill breaches only in the longest prompt band, constrain context or tune prefill scheduling. Thresholds and sample counts must be chosen from the service's own error budget.

Test at realistic arrivals, including bursts, and count goodput: requests satisfying both TTFT and output-stream requirements. Raise offered load until the chosen TTFT percentile crosses its target, then leave operational headroom for failures and traffic variance. When matching ClusterBid capacity to the result, provide the model build, prompt distribution, retrieval configuration, concurrency, and goodput threshold rather than a peak tokens-per-second number.

The budget has limits. Distributed traces add overhead and sampling can miss rare paths. Client clocks and mobile networks add noise. Cache state changes retrieval and prefill behavior, while downstream answer quality may constrain token reductions. Revalidate after model, tokenizer, index, reranker, prompt template, scheduler, or region changes. The goal is not permanent numbers. It is an accounting system that makes the next regression attributable.

vLLM 0.8.5 Production Metrics

W3C Trace Context Recommendation

FP4 Inference on Blackwell Cost Guide 2026: 8x Claims vs Production Reality

A/B Testing for AI Models: Production Inference Experimentation Framework