Start with the communication shape, not the rank count
Tensor parallelism divides operations inside a layer. Its ranks exchange partial results repeatedly as forward and backward computation moves through that layer. Pipeline parallelism divides the model by depth, so neighboring stages exchange forward activations and backward activation gradients at stage boundaries. Data parallel ranks perform the same model computation on different samples and synchronize gradients or parameter shards. Those patterns are not interchangeable, even when every layout uses the same number of GPUs.
The practical rule is to place the group with the most frequent, least hideable communication inside the fastest connectivity domain that can hold it. In many transformer training layouts that group is tensor parallel. This is a hypothesis to verify, not a universal law. Sequence length, microbatch size, gradient bucketing, expert routing, overlap, and framework implementation can change which axis is most sensitive. Memory feasibility also comes first: a smaller tensor-parallel degree is useless if a layer no longer fits.
Draw the boundaries the hardware actually exposes
Inventory each GPU, its CPU socket, PCIe path, local NIC, and direct high-speed GPU links. NCCL discovers PCI topology through sysfs. Its topology dump records the detected system topology, while a separate graph dump, when supported by the deployed NCCL version, records communication graphs. Treat a server label as an administrative boundary, not proof of uniform GPU-to-GPU paths. A node may contain one fully connected NVSwitch domain, several smaller high-speed link domains, or GPUs separated by CPU root complexes.
Record the rank-to-device mapping alongside the topology. A topology diagram without rank placement cannot explain an unexpected cross-boundary collective. Conversely, a rank list without physical paths hides why two apparently adjacent ranks behave differently. Preserve GPU and NIC PCI addresses, hostname, driver and NCCL versions, visible devices, launcher ordering, and the topology dump for each tested layout.
| Parallel axis | Typical repeated traffic | Initial placement preference |
|---|---|---|
| Tensor parallel | Layer-internal reductions and gathers | Keep inside the fastest GPU fabric domain |
| Pipeline parallel | Activations between neighboring stages | Cross a slower boundary only after measuring stage traffic |
| Data parallel | Gradient or parameter synchronization | Allow broader span when bucketing and overlap make it tolerable |
Build candidate rank grids before launching a long run
Suppose an illustrative allocation has 16 GPUs arranged as two eight-GPU fast domains. A feasible model needs tensor parallel size 4 and pipeline parallel size 2, leaving data parallel size 2 because 4 x 2 x 2 = 16. Candidate A forms every four-rank tensor group within one fast domain and lets the pipeline connection cross domains. Candidate B interleaves ranks so each tensor group spans both domains. Both satisfy the arithmetic, but they create different traffic paths.
The original Megatron-LM work describes intra-layer model parallelism as a small set of communication operations and treats it as complementary to pipeline parallelism. That supports examining the axes separately, but does not prove that Candidate A wins on every system. Generate and inspect the actual groups. Confirm that each tensor group is contained in one intended domain and that pipeline neighbors cross only where planned. A scheduler allocation can change hostname order, so never infer placement from rank numbers alone.
Measure the fabric and the workload at two levels
First isolate paths with a communication test. Run the same collective, rank count, data types, message sizes, warmup, and iteration count for an in-domain group and a cross-domain group. NCCL tests reports algorithm bandwidth and bus bandwidth, and supports controlled size sweeps. The comparison reveals a path penalty without claiming that the microbenchmark predicts full training throughput.
Then run a short, repeatable training window for each feasible layout. Keep model shape, sequence length, global and microbatch sizes, precision, accumulation, checkpointing, and input source fixed. Capture median step time after warmup, a tail percentile, GPU duty cycle, and communication timing if the framework exposes it. Pipeline layouts also need stage timing and bubble evidence, because a communication-friendly map can still lose to poor stage balance.
Use a decision rule tied to useful work
Set the rule before seeing the result. One workable rule is: accept a cross-boundary tensor group only if its median steady-state step time is within 3 percent of the contained layout, its p95 step time is within 5 percent, and neither layout changes the effective global batch or numerical behavior. The thresholds are policy choices, not hardware constants. Tighten them for expensive long runs and widen them only when measurement noise has been quantified.
For an illustrative calculation, if the contained layout has a 1.20 second median step and the cross-boundary layout has 1.29 seconds, the increase is (1.29 - 1.20) / 1.20 x 100 = 7.5 percent. It fails the illustrative 3 percent rule. If moving the boundary forces a higher pipeline degree, include bubble cost and memory headroom in the same comparison rather than optimizing communication in isolation. Actual results vary with model, software, topology, and load.
Diagnose a losing layout in a fixed order
Begin by proving the intended ranks landed on the intended GPUs. Next compare the topology dump and selected network interfaces. Then reproduce the gap with the smallest relevant collective and message range. Only after topology and path selection are confirmed should you inspect algorithm choices, channel counts, overlap settings, or framework scheduling. This sequence avoids masking a placement error with tuning knobs.
If the collective gap is large but training is unchanged, computation or overlap may be hiding communication. If the collective result is similar but training diverges, inspect pipeline balance, data loading, CPU affinity, and synchronization boundaries. If only the tail grows, look for competing traffic and per-link error or congestion counters. Each outcome narrows the next experiment without turning correlation into a claimed cause.
Know when the placement rule is insufficient
Topology-aware grouping cannot repair a fabric with unstable links, an oversubscribed uplink, or contention from other tenants. Nor does it select the best parallelism degrees by itself. Larger tensor groups may reduce per-GPU memory while making matrix fragments smaller and communication more prominent. Pipeline parallelism may reduce cross-boundary frequency but add bubbles or uneven stage memory.
Treat the final map as workload-specific evidence with a validity envelope: model revision, sequence and batch shape, node type, software versions, and concurrency conditions. When sourcing capacity, ClusterBid can help match an allocation to these measured topology and fabric requirements, but the acceptance test should still run on the delivered nodes. Recheck after scheduler, firmware, driver, NCCL, or physical topology changes.