Building an Inference SLO from Queue and Decode Metrics

An inference SLO should begin with what a user receives, not what the GPU reports. For a streaming text service, a successful request might mean an admitted request returns a valid first token within a threshold, continues without an excessive inter-token gap, and reaches a valid terminal status. For a non-streaming model, success may instead require a complete response before one deadline. Define separate service classes when interactive and batch traffic have different expectations.

State the population explicitly: production requests at the public serving boundary, excluding documented synthetic checks and user cancellations before admission, for example. Decide how to treat rate-limited requests, invalid inputs, dependency failures, client disconnects, and server cancellations. These choices change the numerator and denominator, so they belong in the SLO document and in versioned metric code, not in a dashboard footnote.

Measure the primary latency from the edge or client-visible server boundary. Internal server duration omits load balancers, network transit, retries, and response delivery. It remains valuable for decomposition, but it is not interchangeable with user-perceived time. Assign a request ID and monotonic timestamps at admission, queue entry, execution start, first response byte or token, each streamed chunk if sampled, and terminal completion.

Use one authoritative duration definition across releases. A retry that eventually succeeds may still have exposed extra delay, so decide whether the logical request or each attempt is the SLI unit. For streaming, distinguish time to first token, inter-token gap, and total completion time. A short first-token time cannot compensate for a stalled decode, and a fast decode cannot repair a long queue wait.

Queue duration measures admitted time waiting for scheduling or a model instance. Decode measures generation work after the relevant execution boundary. Triton, for example, exposes request, queue, input, inference, and output durations, but metric semantics depend on server and backend. Verify whether cached requests, dynamic batches, preprocessing, and streaming phases are included before naming a panel or alert.

Build a conservation check: end-to-end server time should approximately reconcile with queue, input, compute, output, and known framework overhead for the same request population. Large unexplained residuals reveal mismatched boundaries or missing stages. Do not add independently aggregated p99 values; the request at p99 queue time is generally not the request at p99 decode time, so their sum is not a valid p99 total.

Suppose, illustratively, an interactive class has an objective that 99.5 percent of valid admitted requests receive a first token within 2 seconds over 28 days. If the period contains 4,000,000 eligible requests, the allowed bad-request fraction is 1 - 0.995 = 0.005, giving 4,000,000 x 0.005 = 20,000 bad requests. A timeout, internal error, or first token after 2 seconds consumes the same request-based budget under this definition.

Assume 12,000 requests miss from queue delay and 3,000 fail before producing a token. The union is 15,000 only if the groups do not overlap; if 1,000 requests appear in both, unique bad requests are 12,000 + 3,000 - 1,000 = 14,000. These figures are illustrative, not production measurements. Count once at the request boundary, then attach reason labels for diagnosis so overlapping internal symptoms do not spend the budget twice.

Percentiles must be calculated over the intended request population and time window. Prometheus documents that precomputed summary quantiles cannot meaningfully be aggregated across replicas, while compatible classic histogram buckets can be summed and passed to histogram_quantile. Choose bucket boundaries densely around the SLO threshold so interpolation error is acceptably small there, and preserve counts for low-volume services.

Never average pod p99 values or average daily p99 values to claim a fleet monthly p99. Aggregate histogram buckets across the selected replicas first, then calculate the percentile. Better yet, for an SLO threshold, compute the direct fraction of requests at or below that boundary from counters or histogram buckets. Percentiles remain helpful for exploration, but threshold compliance maps cleanly to good and total events.

A latency chart without load invites the wrong story. Display eligible arrival rate, admitted rate, successful completion rate, input-token distribution, requested output-token distribution, active sequences, batch size, queue depth and age, ready replicas, and errors alongside the SLI. A falling p99 during an outage may simply mean slow requests timed out before being observed by the successful-latency metric.

Apply a minimum-traffic rule to short-window alerts while keeping the long-window SLO exact. Segment diagnosis by model revision, service class, region, and response-length band, but avoid defining so many SLOs that each has little traffic. Labels such as individual request ID must stay in traces or logs, not metric dimensions. Watch cardinality and retention costs as part of the instrumentation design.

When the first-token SLO burns quickly, first verify total and bad-event counters at the user boundary. Second, split bad events into errors and late successes. Third, compare queue age and queue duration with decode or compute duration. Fourth, check request mix and ready capacity. Queue growth with stable decode suggests insufficient capacity or scheduling blockage; stable queue with slower decode suggests model, kernel, batching, or workload-mix change. Rising errors require their own branch before scaling.

Adopt a measurable response rule such as: if the one-hour bad-event rate exceeds 14 times the allowed long-window rate and request volume exceeds the alert floor, page and freeze nonessential rollouts; if queue contributes more than half of late-request elapsed time, investigate admission, scheduler, and capacity first. The multiplier and contribution boundary are illustrative policy choices. Validate them against historical incidents before attaching automation.

Review the SLO with product, serving, and platform owners. Store its event definition, exclusions, thresholds, windows, data source, query, ownership, and error-budget response in version control. Google SRE's example policy illustrates that an error budget needs an agreed action when exhausted; the exact freeze or escalation policy should fit the organization rather than being copied unchanged.

Audit sampled traces against counters after instrumentation changes, and run synthetic requests to verify timestamps without counting them in the production denominator. The limitation is that server metrics cannot fully represent users behind slow networks or disconnected clients unless measurement reaches that boundary. Keep queue and decode metrics as powerful diagnostic components, while the SLO itself remains anchored to a clearly defined user-visible outcome.

NVIDIA Triton Inference Server Metrics

Prometheus Histograms and Summaries

Google SRE Workbook Example Error Budget Policy

FP4 Inference on Blackwell Cost Guide 2026: 8x Claims vs Production Reality

A/B Testing for AI Models: Production Inference Experimentation Framework