Write the test before access begins
A useful trial is a time-boxed technical acceptance exercise, not an open-ended benchmark session. Before credentials arrive, name the exact workload, container digest, dataset snapshot, command line, precision mode, batch or sequence shape, and number of GPUs. Record the driver, runtime, framework, collective library, and any compilation settings. This freezes the question being asked. A result from a smaller model, shorter context, or different precision may be interesting, but it does not answer whether the offered capacity can run the intended job.
Separate gates from observations. Gates are conditions that can stop a reservation, such as completing a representative training slice without an error and meeting a throughput floor. Observations explain the result, including GPU activity, host memory pressure, data-loader stalls, and collective time. Agree on repetitions and warm-up before seeing results. Otherwise a convenient run can quietly become the official run. This is an operational plan, not legal language, and commercial or contractual terms need separate review by the appropriate people.
| Item | Freeze before trial | Evidence |
|---|---|---|
| Workload | Image digest, code revision, inputs | Manifest and command |
| Success | Threshold and allowed variance | Timestamped result |
| Environment | Driver, runtime, topology | Inventory output |
| Exceptions | Permitted substitutions | Written change log |
Prove which machine was tested
An accelerator label alone does not identify an eight-GPU system. Capture GPU model and memory, GPU count, CPU model and socket count, installed RAM, local storage layout, network interfaces, driver version, and the GPU-to-GPU and GPU-to-NIC topology. NVIDIA documents that nvidia-smi topo -m reports connection paths plus CPU and memory affinity. Preserve its raw output with the trial record, together with GPU UUIDs or PCI bus IDs. A screenshot is less useful than parseable text because later comparisons need complete rows. NVIDIA also warns that nvidia-smi text output is not guaranteed to remain backward compatible, so retain the tool and driver versions or normalize fields at collection time.
Also record whether the trial uses a whole node, a virtual machine, a container, or a partitioned device, and whether another tenant can share the host. Run an idle baseline before the workload and note background GPU processes, filesystem traffic, and CPU load. These details do not prove interference, but they reveal plausible explanations for unstable results. If the eventual reservation differs in CPU, RAM, storage, NIC, topology, tenancy, driver, or firmware, treat it as a different configuration requiring an explicit equivalence decision.
Measure workload outcomes before utilization
Choose an application metric that maps to delivery: accepted samples per second, tokens per second at a fixed sequence mix, completed images per minute, or time to a fixed training step. Pair it with correctness, such as loss progression, evaluation output, numerical tolerance, or output hash where deterministic behavior is expected. Throughput without correctness is not acceptance. Report a distribution from repeated measured windows rather than the best moment, and exclude warm-up only according to the written procedure.
Utilization is supporting evidence. NVIDIA DCGM describes SM activity and DRAM activity as interval averages and cautions that a high value alone does not prove a workload is compute-bound or memory-bound. Collect several related metrics alongside application throughput. PyTorch Profiler can expose operator and device activity, shapes, memory, and traces, but profiling adds work and can alter timing. Use an unprofiled run for the acceptance number and a separate profiled run for diagnosis, keeping their purposes distinct.
Exercise one controlled failure path
A reservation may fail operationally even when a clean benchmark is fast. Test one recovery path that matters to the workload: resume from a checkpoint, replace a failed worker, restart a container, or reattach storage. Measure time from interruption to useful progress, not merely process launch. Verify the resumed step, optimizer state where applicable, data position, and resulting metric. Do not deliberately reset devices or disrupt shared infrastructure without the operator's approved procedure.
Run available health checks before and after the workload. NVIDIA states that DCGM diagnostics can exercise memory, compute, PCIe, NVLink, power, and thermal paths, while also warning that diagnostics can consume resources and conflict with applications. A passed diagnostic is evidence about that invocation, not proof that every future job will succeed. A failure can originate in permissions, libraries, topology, cooling, or workload interference as well as the accelerator, so preserve error codes and context rather than reducing the record to pass or fail.
Assign an owner to every artifact
The trial plan should say who captures each artifact, where it is stored, and when both sides can inspect it. The capacity operator can supply node inventory, topology, diagnostic output, maintenance events, and platform logs within its access boundary. The workload team can supply the image digest, input manifest, commands, application logs, correctness results, and profiler traces. A shared run index should connect them using UTC timestamps and a run identifier. Redact credentials and sensitive payloads before sharing.
Evidence ownership is also about reproducibility after access ends. Keep raw outputs, not only a slide summary, and attach hashes to immutable inputs and images. Record failed and aborted attempts because they affect operational confidence. If a provider cannot disclose a low-level log, ask for an agreed derived signal such as a timestamped incident classification. Do not infer silence means no platform event. The limitation is fundamental: each party sees only part of the system, so conclusions should identify what remains unobserved.
Apply a decision rule without renegotiating it
Consider an illustrative acceptance rule for a four-hour trial: three identical 30-minute measured runs must each exceed 920 accepted samples per second, correctness must remain within the workload owner's stated tolerance, no run may have an unhandled accelerator error, and checkpoint recovery must restore useful progress within 12 minutes. Suppose results are 947, 936, and 901 samples per second, with correct outputs and a nine-minute recovery. The arithmetic minimum is min(947, 936, 901) = 901, below 920, so the configuration does not pass as written.
That failure is a prompt to classify, not to bargain with the metric. Repeat only if the plan already allows a rerun for a documented environmental fault. Otherwise change the requirement or configuration openly and run a new trial version. Compare later offers against the measured GPU count, topology, memory, storage, and network requirements, and keep the evidence packet as the basis for that comparison. Actual application performance varies with software, inputs, system configuration, and contention; a short trial reduces uncertainty rather than eliminating it.