AI Model Deployment Strategies: Blue-Green and Canary Deployments for GPU Inference Serving with Zero Downtime

Kubernetes rolling updates work well for stateless microservices. For GPU inference serving they create problems. A rolling update that replaces pods one at a time leaves the new pods loading model weights into VRAM while the old pods serve traffic. During this window both memory bandwidth and PCIe bandwidth are contended, raising latency for in-flight requests.

GPU inference servers also require model weight initialization, kernel compilation caching, and KV cache warmup before they can serve at full throughput. A pod that appears healthy to the readiness probe may deliver 3x worse latency for the first 60-120 seconds. Standard deployment strategies do not account for this thermal soak period.

Blue-green deployment maintains two identical inference environments. The blue environment serves production traffic while the green environment is provisioned, warmed, and validated. Traffic is switched at the load balancer or ingress layer when green passes health checks at full production load.

For GPU clusters this means provisioning a separate set of inference pods with dedicated GPU allocation. The cost is double the GPU capacity during the transition window. A cluster running 8x H100 SXMs needs another 8x H100 SXMs for green. Typical transition windows run 5-15 minutes depending on model size and weight loading speed.

The advantage is instantaneous rollback. If the new model version regresses on latency or accuracy, traffic shifts back to blue in one DNS TTL. No model weights need to be reloaded. This makes blue-green the preferred pattern for safety-critical inference where any degradation costs more than the spare GPU overhead.

Canary deployments route a small percentage of traffic (typically 1-5%) to a new inference server while the majority continues hitting the stable version. Traffic weight increases in stages as the canary accumulates evidence of correctness and latency stability.

The GPU-specific challenge is that a 5% canary still needs a full GPU allocation for the new model version. You cannot run a model on 5% of a GPU. The minimum unit is one GPU for the inference server. For small models (7B-13B) that fit on a single GPU, this means 2 GPUs total during the canary period. For large models requiring tensor parallelism across 8 GPUs, you need 16 GPUs.

Implementation patterns include using Envoy or Istio weighted routing at the request level, with separate service entries for stable and canary inference services. The canary service runs its own vLLM or Triton instance on dedicated GPUs with identical topology placement to ensure representative latency measurements.

Cold start is the dominant source of deployment failures for GPU inference. Loading a 70B parameter model in FP16 requires 140GB of VRAM writes. Over PCIe Gen5 at 128 GB/s this takes roughly 1.1 seconds for weight transfer. Kernel compilation for CUDA graphs adds another 5-30 seconds depending on model complexity. The KV cache starts empty, so the first 50-100 requests incur higher latency until prefix caching warms.

Warm start strategies include pre-warming the new environment with synthetic traffic before admitting real requests. A production pattern is to send dummy requests matching the production request distribution for 30-60 seconds while monitoring P50 and P99 latency convergence. The environment is marked healthy only when latency stabilizes within 10% of the baseline.

A concrete implementation using vLLM starts with two Kubernetes services: inference-blue and inference-green. Each runs a StatefulSet with GPU resource requests and node affinity for the same GPU type. The ingress controller (nginx-ingress or contour) points to the active service. Deployments script the sequence: provision green, run a load test with a synthetic dataset, compare latency distributions, and switch the ingress upstream.

For Triton Inference Server deployments, the model repository must be versioned and the canary policy configured through Triton's model configuration API. Triton's built-in model version policy supports canary traffic splits at the model level, which avoids the need for a separate service mesh. The GPU memory overhead is identical since Triton loads each model version into separate CUDA contexts.

Key metrics to monitor during any GPU deployment: P50 and P99 inference latency, request throughput (tokens/second), GPU memory utilization (should stay flat), and error rate. A sustained P99 latency increase above 15% is grounds for automatic rollback regardless of other metrics.

Pre-warm the new inference server with synthetic traffic matching your production request distribution. Validate latency at the P50, P95, and P99 percentiles before shifting traffic. Confirm GPU memory utilization is stable and within 90% of the expected profile.

Configure your load balancer or service mesh for session affinity if stateful inference patterns (like conversational context) are in use. Set up automated rollback triggers on latency, error rate, and throughput regression. Document the expected cost of spare GPU capacity for the transition and build it into the deployment budget.

Blue-green and canary deployment patterns require extra GPU capacity during transitions. The safest approach is to over-provision your inference cluster by 20-30% to handle deployment overhead without affecting production throughput. This extra capacity also serves as headroom for traffic spikes and can be released back to the pool when not in use.

On ClusterBid, you can reserve additional GPU nodes dynamically for deployment windows and release them when the transition completes. This avoids paying for idle deployment overhead 24/7 while maintaining the ability to spin up green environments on demand.