Retries create a second workload that dashboards can miss
A structured-output endpoint may return an HTTP success while failing JSON parsing, schema validation, or an application rule. If the client silently asks the model again, one user request becomes two or three inference attempts. Traffic dashboards that count only external requests show stable demand, while serving dashboards show higher token volume, queueing, and GPU time. The gap can look like a capacity regression even though retry expansion caused it. A retry storm can also become self-reinforcing: extra attempts lengthen queues, client deadlines expire, and timed-out callers submit replacements. That possibility should be tested from identifiers and timestamps rather than inferred from utilization alone.
Give each logical request one durable identifier and each attempt a child identifier. Record the requested schema version, decoding mode, prompt and output tokens, validation result, semantic acceptance result, retry reason, and GPU-visible timing. The key distinction is between transport success, syntactic JSON validity, schema validity, and useful acceptance. Combining these outcomes into one success counter makes the retry source impossible to localize.
Measure first-attempt success and eventual success
First-attempt schema success is the share of logical requests whose initial response validates against the intended schema. Eventual schema success is the share that validates within the retry limit. Add semantic acceptance because schema-conforming nonsense is still unusable. Then calculate retry expansion as total inference attempts divided by logical requests, and compute amplification separately for output tokens and GPU-seconds.
An endpoint can report 99 percent eventual schema success while consuming far more capacity than expected. If only 85 percent succeeds on the first attempt and every failure retries once, expected attempts are 1.00 + 0.15 = 1.15 before considering failures on the second attempt. That 15 percent expansion is invisible in a metric that reports only the final response. Segment these rates by schema, model version, prompt template, and finish reason.
Constrained generation and application retries spend work differently
Application-level repair allows unconstrained token selection, then parses or validates the completed text. Its cost appears when failed outputs are discarded and the prompt is run again, sometimes with an error message appended. Constrained generation instead limits valid next tokens according to a grammar or schema during decoding. The llama.cpp project documents conversion of a subset of JSON Schema to its GBNF grammar format, which is an important compatibility boundary rather than a claim of universal schema support.
Constraint processing is not free. Schema compilation, token filtering, branching structure, and backend implementation can affect time to first token or decode speed. OpenAI's 2024 Structured Outputs announcement distinguishes JSON validity from adherence to a supplied schema and describes constrained decoding as part of its approach. Neither that announcement nor an open-source grammar mechanism supplies a portable overhead number for a different model and stack. Measure both paths on the schemas and traffic shapes actually served.
| Approach | Capacity cost to measure | Correctness gap to retain |
|---|---|---|
| Unconstrained plus retry | Discarded tokens, repeated prefill, added queue work | Later attempts may still fail schema or semantics |
| Constrained decoding | Schema compile, token-mask, and decode overhead | Valid structure does not guarantee correct values |
| Constrained plus semantic retry | Constraint overhead plus repeated accepted-format attempts | Application rules can still reject valid JSON |
Worked example: a small failure rate consumes a large share
Consider an illustrative hour with 12,000 logical extraction requests. The first-attempt schema success rate is 88 percent, so 1,440 requests retry. Of those, 80 percent succeed on attempt two, leaving 288 requests that make a third and final attempt. Total attempts are 12,000 + 1,440 + 288 = 13,728. Retry expansion is 13,728 / 12,000 = 1.144, or 14.4 percent more attempts than the external request count suggests.
Assume, illustratively, a first attempt uses 1.8 GPU-seconds, a second uses 2.0 because the repair prompt is longer, and a third uses 2.2. Total GPU work is 12,000 x 1.8 + 1,440 x 2.0 + 288 x 2.2 = 25,113.6 GPU-seconds. Without retries it would be 21,600 GPU-seconds, so compute amplification is 16.27 percent. These are planning examples, not benchmark results; actual overhead depends on the schema, model, engine, and response lengths.
Follow the attempt chain before adding capacity
Use a fixed diagnostic sequence. First, reconcile gateway logical requests with inference-server attempts. Second, chart first-attempt schema success and semantic acceptance by schema version. Third, attribute extra attempts, input tokens, output tokens, and GPU-seconds to retry reason. Fourth, inspect finish reasons such as length limits separately from malformed structure. Fifth, replay a representative sample with retries disabled, then with the production policy, while holding offered logical load constant.
Set a measurable gate: investigate retry behavior whenever attempt expansion exceeds 1.05 for two consecutive observation windows or when retries consume more than 10 percent of GPU-seconds at peak. Those thresholds are illustrative and should reflect the service's margin and reliability objective. Do not buy capacity for the observed peak until the team can state what fraction is first attempts, retry work, and reserve. ClusterBid can help match capacity after that decomposition, not replace it.
Choose the path from local evidence and preserve escape hatches
Run the same fixed corpus through unconstrained generation with the current retry policy and through each supported constrained backend. Report first-attempt schema success, semantic acceptance, p95 time to first token, p95 end-to-end latency, accepted responses per GPU-hour, and schema compilation failures. Warm and cold schema cases should be separate. A decision rule can then require the selected path to meet the semantic target and increase accepted responses per GPU-hour without breaching the latency objective.
There are honest limitations. A constrained decoder can guarantee only the structural language it implements, not factual correctness or business validity. A benchmark corpus may underrepresent rare schemas, deeply nested arrays, or adversarial strings. Application retries may still be necessary for transient transport failures and semantic rejection. Keep a bounded retry budget, expose exhaustion as a distinct outcome, and alert on shifts in first-attempt success so that a quality change cannot quietly become a capacity purchase. Preserve the rejected payload category without logging sensitive content, because malformed syntax, missing required fields, and prohibited values point to different remedies.