Every parallel strategy buys something and communicates something
The default is the smallest single replica that fits. Parallelism is a memory or latency tool; data-parallel replicas are the ordinary throughput tool once a replica works.
| Strategy | Split | Communication | Use when | Failure mode |
|---|---|---|---|---|
| Data parallel (DP) | Requests/batches across full model replicas | Little for independent inference; gradients in training | One replica fits; need more throughput | Duplicated weights, fragmented batches and caches |
| Tensor parallel (TP) | Matrices/heads inside every layer | Collectives every layer | Model/KV does not fit one GPU; fast intra-node links | Cross-node collectives or tiny per-rank GEMMs dominate |
| Pipeline parallel (PP) | Contiguous layer groups | Activations between stages | Need to extend beyond one link domain | Bubbles, imbalance, higher single-request latency |
| Expert parallel (EP) | MoE experts across ranks | Token dispatch/return, often all-to-all | Large MoE models | Hot experts and network imbalance |
| Context parallel (CP) | Sequence dimension | Attention-specific exchange | Extreme contexts that cannot fit/process on one rank | Narrower support and communication overhead |
Keep the most frequent communication on the fastest links
Inside GPUHBMTB/s-class local memory
Inside nodeNVLink / Infinity FabricPreferred TP / EP domain
Across nodeInfiniBand / RoCEDP, PP, scale-out collectives
Across zoneOrdinary networkReplicas and asynchronous work
- If one GPU fits and passes, stop sharding.You retain simpler failure, scheduling, and performance behavior.
- Use the smallest TP degree that fits.Keep TP inside one NVLink/NVSwitch or qualified Infinity Fabric domain when possible.
- Cross node boundaries deliberately.Prefer PP between slower domains; cross-node TP only when the fabric and workload prove it.
- Add DP after the replica works.Independent replicas scale throughput and failure isolation, subject to batching and cache locality.
- Pin topology.Schedule ranks into the expected link clique, NUMA domain, NIC, rack, and zone.
Separate prefill and decode only after interference is proven
Disaggregation gives the two phases independent pools and scaling policies, but now the system must move KV state and operate two capacity planes.
| Keep one pool when | Consider two pools when |
|---|---|
| Traffic is modest or homogeneous | Long prompts measurably damage decode tail latency |
| Prompts are short and KV transfer would dominate | TTFT and TPOT need different replica counts or parallelism |
| Either phase would idle in a separate pool | Traffic is stable enough to keep both pools useful |
| Operational simplicity is worth more than a narrow gain | The network moves KV faster than the interference it removes |
Prove that scale buys useful work
- Topology dump matches the intended peer links, NICs, NUMA nodes, and scheduler labels.
- Model heads, layers, experts, quantization groups, and kernel shapes divide across the chosen ranks.
- Per-rank memory is balanced; one stage or expert does not set the limit.
- Communication time, overlap, and bytes per step/layer are measured - not inferred from link marketing.
- Scaling efficiency is reported against one GPU or one node at the same useful workload.
- Failure and restart behavior is tested for a rank, node, NIC, and collective timeout.
- The extra throughput or time-to-result is worth the extra GPU, fabric, power, and idle reserve.
scaling efficiency = throughput at N ÷ (N × throughput at 1)