A server port speed describes only one edge
A quoted 400 Gb/s server port says how fast that port can signal under its configured mode. It does not state how much capacity remains after traffic reaches the top-of-rack switch, how many servers share uplinks, whether paths are balanced, or which other tenants can contend. For distributed GPU jobs, the useful question is how much simultaneous bandwidth the placed ranks can obtain across every boundary they must cross.
Document downlink capacity and uplink capacity separately. NVIDIA's data-center cabling guidance defines top-of-rack oversubscription as host downlink bandwidth divided by network uplink bandwidth. That ratio is a starting point, not a workload guarantee. A nominal 1:1 leaf can still encounter contention elsewhere, while an oversubscribed leaf may be adequate when most communication remains within a rack or peaks do not coincide.
Draw topology boundaries before calculating ratios
Request a logical diagram that identifies GPU nodes, NICs, leaf switches, uplink groups, spine switches, and any additional aggregation tier. Mark link count and configured speed at each boundary. Also record failure domains and routing behavior, because losing one member can change available capacity without changing server port speed.
Give each shared boundary a name such as rack A leaf uplinks or pod 3 spine egress. Calculate a ratio at every boundary rather than publishing one cluster-wide number. Include whether the value is physical installed capacity, capacity after redundancy reservation, or measured usable throughput. If a provider will not disclose device models, it can still provide anonymized link counts, speeds, and sharing groups.
| Procurement field | Required detail | Why it matters |
|---|---|---|
| Server-facing ports | Count, configured speed, bonding, GPU or NIC affinity | Defines potential injection capacity |
| Leaf uplinks | Count, speed, active members, sharing group | Defines first shared ceiling |
| Spine paths | Number of equal-cost paths and failure case | Shows path diversity and reduced-capacity state |
| Placement policy | Ranks per rack and topology awareness | Determines which boundaries a job crosses |
| Test conditions | Concurrent senders, message sizes, direction, duration | Makes simultaneous bandwidth reproducible |
Calculate both the headline and the usable share
Consider an illustrative rack with eight GPU servers, each exposing two 200 Gb/s fabric ports. Aggregate downlink capacity is 8 x 2 x 200, or 3,200 Gb/s. If the leaf has four 400 Gb/s uplinks, aggregate uplink capacity is 4 x 400, or 1,600 Gb/s. The top-of-rack oversubscription ratio is 3,200 / 1,600, or 2:1. These are illustrative planning assumptions, not a description of a provider.
If all eight servers send out of rack at equal rates and the uplinks are the only ceiling, an equal-share upper bound is 1,600 / 8, or 200 Gb/s per server across both ports. That is 100 Gb/s per 200 Gb/s port if traffic divides evenly. Protocol overhead, path hashing, receiver limits, and other traffic can reduce measured application throughput. The calculation also says nothing about traffic that stays within the leaf, so label the assumed traffic matrix.
Require simultaneous bandwidth, not isolated links
A one-server test can show that a port and path operate, but it cannot expose the shared uplink ceiling. Ask for at least three cases: one sender and one receiver, all servers communicating within the rack, and all servers communicating across the rack boundary at the same time. Add the failure case promised by the service, such as one uplink unavailable, if redundancy is part of the offer.
Specify direction and pattern. Unidirectional streams, bidirectional exchange, all-reduce, all-to-all, and incast stress different resources. Record message sizes, rank count, placement, duration, warmup, software versions, and background load. Report per-node distribution as well as aggregate throughput. An aggregate can meet target while one hash bucket or one server performs poorly.
Define an acceptance envelope around the job
Convert the application profile into simultaneous requirements and calculate each direction separately. Suppose an illustrative 16-node job places eight nodes in each of two racks and requires each node to transmit 220 Gb/s to the opposite rack during its communication phase. Eight senders in either direction demand 8 x 220, or 1,760 Gb/s in that direction. A documented full-duplex boundary with 1,600 Gb/s available in each direction cannot satisfy that assumption without queueing or scheduling, even though every server has 400 Gb/s of port capacity. Adding all 16 senders into one 3,520 Gb/s figure would incorrectly combine opposite directions unless the quoted boundary capacity were also defined as a bidirectional aggregate.
A measurable rule could require each node to sustain at least 90 percent of the agreed per-node target while all intended peers are active, with the slowest node no lower than 85 percent, over a 10-minute test. Those percentages and duration are illustrative policy choices, not vendor standards. Define separate limits for normal and reduced-capacity states. Reject results that omit placement or concurrent sender count because they cannot be mapped back to the demand calculation.
Keep design capacity separate from observed throughput
A topology sheet describes possible ceilings. A test describes behavior under one traffic matrix and software stack. Store both. If measured throughput is below the calculated boundary, investigate endpoint limits, transport configuration, route imbalance, receiver contention, and protocol overhead before declaring hidden oversubscription. If it reaches the ceiling, that still does not prove every pattern will distribute equally.
NVIDIA's older DGX networking best-practices document illustrates that oversubscription can be introduced at the server-to-leaf or leaf-to-core boundary and that these choices affect bandwidth and latency. The enduring procurement lesson is to name the boundary. Avoid presenting a ratio without its numerator, denominator, redundancy state, and traffic direction.
Make the capacity record updateable
Attach the diagram, boundary table, calculations, raw test output, and acceptance result to the purchase record. Add triggers for retest: a leaf replacement, uplink-count change, new rack, routing-policy change, NIC firmware upgrade, or altered placement policy. Date every artifact. Compare capacity offers against the same measured envelope when each seller supplies these fields rather than only accelerator and port labels.
The honest limitation is that disclosure and preproduction tests cannot guarantee performance under every future tenant mix or failure. Hash collisions and burst synchronization can create tails even when average demand fits. Reduce that uncertainty with placement commitments, telemetry access, concurrency tests, and a defined remedy or revalidation process. The goal is not a perfect ratio. It is a traceable statement of who shares which links, under what conditions, and what the buyer actually measured.