Translate the launch plan into arrivals
Launch traffic should enter the capacity plan as an arrival curve, not one daily request total. Estimate requests per second by five minute interval, then split them by model, region, input-size band, output-size band, and streaming behavior. Record the source of each multiplier, such as invited users, notification batches, retry policy, or marketing schedule. Keep base, expected, and stress curves rather than hiding uncertainty in one number.
An illustrative expected peak is 24 requests per second with an average end-to-end time in the serving system of 8 seconds under a stable workload. Little's Law gives 24 multiplied by 8, or 192 requests in the system on average before a safety margin. Suppose a workload-shaped test also shows that one warmed worker sustains 1.5 requests per second and 12 requests in the system at the required latency. Both checks give an arithmetic floor of 24 divided by 1.5, or equivalently 192 divided by 12, which is 16 workers. Adding 25 percent to that floor produces 20 ready workers. These numbers are hypothetical and illustrative, not performance claims; use a percentile-aware load test to validate the fleet because averages do not bound tail latency.
Define the unit of ready capacity
A provisioned GPU is not necessarily launch capacity. Count a worker only after the node is available, the image and model are loaded, warmup has completed, health checks pass, and a representative request meets the service objective. NVIDIA Triton documents model warmup that holds a model unready until configured warmup requests complete. Equivalent behavior in another server should be verified rather than assumed.
Create a readiness score with measurable components: required worker count, warm success rate, model checksum match, request p95, error rate, and spare queue capacity. The launch gate might require 20 of 20 workers ready for 30 continuous minutes, zero model-load failures, and p95 below the internal threshold under a rehearsal load. Thresholds are illustrative and must come from the product objective and observed variance.
Prewarm before the irrevocable event
Work backward from the first user-facing event. If the measured p99 time from capacity request to ready worker is 18 minutes, do not rely on a five minute autoscaling reaction. Bring the baseline online early enough to absorb provisioning variance, image pulls, model loading, and retries. Stagger starts so a bad artifact does not fail every worker simultaneously, then freeze unrelated infrastructure changes.
Autoscaling remains useful for the tail, but its signal should reflect demand before latency collapses. Kubernetes HPA can scale on custom or external metrics, and it handles startup readiness specially. Queue depth, admitted concurrency, or tokens waiting may lead GPU utilization for a bursty service. Set the maximum replicas high enough for the stress curve, and verify that quotas, IPs, volumes, and GPU supply allow the controller to reach it.
Load-test the decisions, not just the endpoint
Rehearse the expected curve, a faster ramp, a sustained peak, and a dependency slowdown. Preserve the real mix of short and long requests. Measure arrival rate, admission rate, time to first response, completion latency, errors, queue age, cancellations, worker startup time, and cost per successful unit. A test that generates only uniform short prompts can validate plumbing while missing the launch's memory and batching behavior.
The pass rule should be binary enough for an incident lead to use. For example: expected load passes if the service holds its latency and error objectives for 30 minutes with at least 20 percent ready headroom; stress load passes if overload controls preserve priority traffic and recovery begins within 10 minutes after arrivals fall. These are illustrative gates. Document every exception with an owner and expiry.
Prepare a fallback ladder with named triggers
Fallbacks must be ordered before launch pressure arrives. A sensible ladder might disable nonessential batch jobs, reduce maximum output length, tighten per-user concurrency, route eligible traffic to a smaller model, pause new sessions, and finally show a controlled busy response. Each step needs an owner, a reversible command or configuration change, a user impact statement, and the metric that triggers it.
Use sustained signals rather than intuition. An illustrative trigger for step one is queue age above 4 seconds for 3 consecutive minutes while ready capacity exceeds 90 percent admission occupancy. Step two could follow if queue age exceeds 8 seconds or error budget burn crosses the declared launch threshold. Do not wait for out-of-memory failures. The exact values must be derived from user-facing deadlines and tested control behavior.
| Condition | Decision | Owner evidence | Recovery test |
|---|---|---|---|
| Ready headroom below 20% | Pause nonessential GPU jobs | Freed worker count | Headroom above 25% for 10 min |
| Queue age above tested trigger | Tighten output or concurrency limit | Queue and rejection metrics | Queue below recovery threshold |
| Primary pool impaired | Route only validated eligible traffic | Fallback error and latency metrics | Primary canary passes |
Run the launch on a clock
At T minus 120 minutes, verify provider allocation, quotas, model artifacts, dashboards, paging routes, and fallback permissions. At T minus 60, complete warmup and a short synthetic check from each region. At T minus 15, compare ready workers with the gate and declare go, hold, or reduce scope. After launch, record decisions at fixed intervals so the team sees the same capacity picture.
Assign one capacity lead, one application lead, one incident commander, and one communications owner. The capacity lead can add workers and halt background demand; the application lead owns admission and fallback controls. Avoid shared ambiguous authority. ClusterBid can help match extra supply to the measured worker profile, region, and time window, but the go decision still depends on verified ready capacity rather than a purchase confirmation.
Recover deliberately after the peak
Do not scale down on the first quiet interval. Confirm that notification waves, retries, and regional peaks have passed, then remove temporary restrictions in reverse order. Keep enough workers ready to absorb retry traffic while inspecting error cohorts. A recovery rule could require queue age below target, error rate within budget, and at least 25 percent headroom for 20 minutes before each fallback is reversed.
Within one business day, reconcile forecast versus actual arrivals, ready-worker yield, startup distribution, fallback activations, failed requests, and excess GPU-hours. The limitation is that a rehearsal cannot reproduce social amplification, provider shortages, or every dependency failure. The runbook manages that uncertainty by defining observations and reversible decisions, not by pretending the forecast is exact.