Orchestration is several control loops, not one product
Each layer owns a different unit: request, model replica, job, pod, device, or physical node. Add a layer only when its problem exists.
Choose the serving engine with the model and feature matrix
| Engine | Strong starting fit | Strengths | Check |
|---|---|---|---|
| vLLM | General open-source baseline | Broad model/hardware support, PagedAttention, batching, cache, quantization and parallelism | Fast-moving compatibility matrix and feature combinations |
| SGLang | Prefix-heavy, structured, agentic and multi-turn programs | RadixAttention, cache-aware execution, structured generation | Benefit depends on real prefix/control-flow reuse |
| TensorRT-LLM | NVIDIA-only performance engineering | Platform kernels/formats, in-flight batching, KV tooling, detailed benchmarking | Greater vendor coupling and build/configuration complexity |
| TGI | Existing deployments already operating it | Mature API, metrics and tensor parallelism | Officially maintenance mode; do not make it the default greenfield choice |
- Verify the exact model architecture.Multimodal, embeddings, structured output, LoRA, speculation, and custom ops can narrow the field.
- Verify the GPU and precision.A format listed by the model is not necessarily accelerated by the engine on that generation.
- Replay one production trace on two viable engines.Pin versions and tune each fairly.
- Select by operations as well as goodput.Startup, upgrades, metrics, failure behavior, and team familiarity belong in the score.
Kubernetes allocates devices; queueing systems allocate opportunity
Kubernetes + device plugin
Exposes GPUs as integer extended resources and places pods. In a heterogeneous fleet, request model, memory class, topology, and sharing mode through labels/affinity - not merely “one GPU.”
GPU Operator
Automates NVIDIA drivers, container toolkit, device plugin, feature discovery, MIG management, and DCGM telemetry on supported combinations. It does not choose your model topology.
Kueue
Admits queued jobs against quota and resource flavors, supports fair sharing, preemption and topology-aware scheduling, then hands admitted pods to the ordinary scheduler.
Volcano
Adds HPC/batch scheduling policy such as gang scheduling, queues, DRF, bin packing, and backfill. Useful when distributed jobs must start together.
Interactive inference. Keep warm Deployments or StatefulSets with explicit minimum capacity, readiness gates, load shedding, and request-level autoscaling signals.
Training and offline batch. Queue whole jobs, reserve the required topology together, checkpoint durably, and allow fair sharing or preemption only where the workload can recover.
Add the next layer when a named failure repeats
| Observed problem | Add or change | Do not add when |
|---|---|---|
| One engine cannot fit the model | Intra-node TP; then PP across nodes if required | A larger-memory single GPU is cheaper and passes |
| Replica routing, multi-model graphs, or custom autoscaling is hard | Ray Serve or an equivalent application orchestrator | One Deployment and service already meets the need |
| Teams fight over queued GPU jobs | Kueue quotas/resource flavors/fair sharing | The workload is always-on interactive inference |
| Distributed jobs deadlock waiting for the rest of the gang | Volcano or gang/all-or-nothing admission | Jobs are independent single-pod work |
| Long prompt prefill ruins decode tails | Tune chunking first; evaluate phase disaggregation | No measured phase interference exists |
| Cold autoscaling misses the SLO | Warm minimum replicas, pre-pulled images/weights, faster initialization | Scale-to-zero latency is acceptable |