Replace performance adjectives with executable tests
An RFQ phrase such as high-performance interconnect is not an acceptance criterion. It lacks a test binary, topology, message size, duration, statistic, threshold, and failure response. A useful requirement says what the supplier must run, on which installed system, under what conditions, and what raw evidence must be delivered. It also separates a technical pass condition from commercial remedies, warranties, or service guarantees that belong elsewhere in the procurement documents.
Name immutable artifacts where possible: container digest, benchmark source revision, build options, configuration files, and output schema. Define the GPU count and placement, node count, processes per node, CPU binding, network interfaces, and whether tests run on an otherwise idle cluster. Identify who supplies test data, who operates the commands, and how both parties witness or sign the final result bundle. The installed configuration is the system under test. Results from a vendor laboratory, a smaller cluster, or a different firmware stack can inform expectations but should not substitute for site acceptance.
Cover correctness, locality, fabric, and sustained workload behavior
Acceptance should progress through layers. Inventory verifies device identities, link widths, firmware, storage, and network interfaces against the bill of materials. Node tests exercise GPU memory, compute, host-to-device paths, local storage, and GPU-to-GPU paths. Fabric tests exercise the actual collective shapes. Finally, a representative workload test catches coordination effects that component checks may miss. NVIDIA DCGM diagnostics provide active tests for memory, PCIe, compute, memory bandwidth, stress, power, and GPU copy paths, but their documentation also says diagnostics do not repair faults or decide replacement eligibility.
For collectives, specify correctness and performance. The official nccl-tests suite checks both and can scan message sizes across a chosen rank count. Require the operations used by the intended software, not only all-reduce by habit. An inference fleet using tensor parallelism may care about small-message latency and all-gather behavior, while data-parallel training may emphasize large all-reduce and reduce-scatter. Report both central tendency and dispersion rather than a single best run.
Define the workload shape before defining the threshold
A result is meaningful only for its shape. State model or synthetic workload version, precision, global and per-device batch, sequence or input dimensions, parallelism layout, warmup, measured interval, checkpoint policy, data source, and compilation state. If the acceptance workload uses generated data to isolate compute, say so. If it reads storage, define the dataset placement and cache state. Otherwise two parties can produce valid but incomparable numbers.
Specify a sustained interval long enough to expose clock, thermal, memory, storage, and congestion effects. Do not define sustained as the best minute inside a run. A practical clause can require a thirty-minute measured window after a stated warmup, with per-minute samples retained and no excluded samples except a predeclared instrumentation failure. The exact duration is a buyer choice. It should reflect the intended workload and facility behavior, not a universal benchmark rule.
Worked example: an acceptance schedule with measurable gates
Assume, illustratively, a 32-node cluster with eight GPUs per node. The RFQ could require inventory on all 32 nodes, active node diagnostics on all 256 GPUs, single-node collective tests on every node, and multi-node tests on four predefined groups of eight nodes plus one 32-node run. For a sustained training proxy, require three runs of 45 measured minutes after 10 minutes of warmup. These quantities are illustrative planning assumptions, not claims about what any specific cluster will achieve.
A measurable rule might read: every correctness test must report zero wrong results; all required links must be active; each node's large-message all-reduce bus bandwidth must be at least 90 percent of the median across all 32 nodes on the first pass; and each sustained run must achieve at least the mutually agreed throughput threshold with a coefficient of variation no greater than 5 percent across five-minute samples. NCCL bus bandwidth is a collective-specific normalization, not a direct measurement of one physical link. If the first-pass median is 300 GB/s and one node measures 255 GB/s, its ratio is 255 / 300 x 100 = 85 percent, so that node fails the hypothetical 90 percent relative gate and is retested after isolation.
| Layer | Required evidence | Example decision |
|---|---|---|
| Inventory | Machine-readable device, firmware, topology, and link records | Exact approved configuration or documented variance |
| Node health | Raw diagnostic results per GPU and node | Zero unexplained failures |
| Collectives | Commands, rank map, errors, time, algbw, busbw | Correct and above absolute or peer-relative floor |
| Sustained workload | Time series plus aggregate throughput | Threshold met for full measured window |
Make exclusions and retests explicit
List what acceptance does not cover. Examples include application accuracy, future software releases, performance under buyer-created oversubscription, external data services, wide-area connectivity, and workloads whose shapes were not supplied. Also identify permitted maintenance states and environmental prerequisites. An exclusion narrows the technical claim; it should not be worded as an interpretation of legal responsibility.
Define retest mechanics before a failure occurs. Require preservation of the failing logs, node identity, timestamps, telemetry, and test inputs. Permit a limited number of retries only after the cause and corrective action are recorded. A rerun that silently discards the first result is not evidence of stability. State whether a replaced node repeats only its local tests or the affected group and full-cluster tests. This keeps the acceptance process reproducible instead of negotiable at each defect.
Keep the technical schedule separate from the guarantee
A benchmark threshold demonstrates capability under stated conditions. It does not by itself promise availability, future workload performance, response time, or a remedy after handover. Put uptime definitions, support response, spare policy, credits, and other commercial terms in the appropriate agreement, reviewed by qualified procurement and legal teams. Cross-reference the technical schedule without asking benchmark prose to carry contractual meanings it cannot define.
No acceptance suite explores every rank placement, simultaneous workload, failure mode, or future library version. The honest goal is bounded evidence. A supplier may help match capacity to the tested node count, topology, and workload envelope, but the buyer still needs to execute acceptance on the delivered configuration. Preserve the final commands, signed result bundle, exceptions, and approved deviations as the baseline for later regression tests.