Start with the measured quantity, not the headline
A claim such as three times faster is incomplete until the numerator and denominator are named. Ask whether faster means samples per second, tokens per second, time to train, time to first token, inter-token latency, or completed requests within a latency target. These quantities answer different buying questions. A system can raise aggregate throughput by batching more work while making an individual request wait longer. A training run can report high device throughput while spending substantial wall time outside the measured loop.
Write the claim as a sentence with blanks: this system completed a named workload, at a named quality threshold, under a named latency or completion condition, compared with a named baseline. If any blank cannot be filled from the provider's material, record it as unknown rather than assuming a favorable value. That simple rewrite turns marketing language into a testable proposition and prevents a percentage from floating free of its operating conditions.
Pin the model, software path, and numerical precision
Record the exact model identity, revision, weights, framework, runtime, compiler settings, driver, and relevant libraries. Model family names are not precise identifiers. Changes in architecture, vocabulary, attention implementation, kernel selection, or graph compilation can alter both work performed and measured speed. For generative inference, distinguish prompt processing from token generation and state the input and output length distributions.
Precision belongs beside the result, not in a footnote. Ask whether computation and weights use FP32, TF32, FP16, BF16, INT8, FP8, or a mixture, and whether sparsity is enabled. Then ask what quality check accompanied conversion or quantization. Lower precision or structured sparsity may be entirely appropriate, but a speed comparison is only informative when both the numerical method and the accepted accuracy or quality boundary are visible.
Make batch and input shape explicit
Batch size, concurrency, sequence length, image dimensions, and input distribution determine which parts of a system are exercised. A large offline batch can keep accelerators full, while an online service may be constrained by queueing and a tail-latency objective. For a language model, averages can conceal long prompts that dominate memory use and prefill time. Request a histogram or a small set of representative buckets rather than one average length.
Use a decision rule: reject direct comparability if batch, concurrency, input shape, or service-level constraint is missing. Mark the result useful only for directional screening if those fields exist but differ from your workload. Call it candidate evidence for purchasing only when the provider can rerun your declared load shape and report repeated completed-work results. This rule is deliberately strict because an unspecified load generator can manufacture an attractive but irrelevant operating point.
Draw the system boundary around every timer
Ask where the clock starts and stops. Does inference timing include host preprocessing, tokenization, transfers, queueing, distributed communication, and output handling? Does training time include evaluation, checkpointing, data loading, and recovery? Note the number and type of accelerators, CPU model, memory, storage, network, topology, power mode, and whether the run used one node or a cluster. A GPU-only kernel number cannot establish application or service throughput.
MLPerf Inference v4.0 is useful as an evidence model because its published results repository exposes submitted results and code, while the versioned benchmark repository defines scenarios, system descriptions, accuracy checks, and audit requirements. A provider does not need to run MLPerf for every evaluation. It should, however, provide an equally legible chain from workload and system configuration to logs and result. The relevant lesson is disclosure, not the use of a particular leaderboard.
Worked example: triage a five-times claim
Consider an illustrative claim: five times higher LLM throughput than a prior GPU. The provider reports 5,000 versus 1,000 tokens per second. Follow-up reveals that the first result uses batch 64, FP8, 512 input tokens, and four accelerators, while the baseline uses batch 8, FP16, 2,048 input tokens, and one accelerator. The arithmetic ratio is 5,000 divided by 1,000, which equals 5, but the conditions changed in at least five material ways.
Classify this as marketing evidence, not a false statement and not a reproducible comparison. It proves only that two differently configured systems produced two reported rates. Ask for per-accelerator results at the same batch, precision, prompt distribution, output length, software revision, and latency constraint, repeated at least three times. If the coefficient of variation for completed throughput exceeds an illustrative 5 percent threshold, investigate warm-up, contention, thermal state, or sampling before using the mean. The threshold is a planning convention, not a universal standard.
Grade evidence by what another operator can repeat
Use three grades. Marketing evidence supplies a headline and perhaps selected configuration facts, but no runnable method or raw output. Comparable evidence fixes workload, quality boundary, system scope, and measurement procedure, yet may rely on a provider-controlled environment. Reproducible evidence adds versioned code or commands, configuration files, logs, run counts, and enough system detail for an independent operator to obtain a result within a declared tolerance.
Do not reward volume of documentation by itself. A long report that omits the baseline configuration remains weak. Conversely, a compact repository with a commit, container digest, workload manifest, system description, raw logs, and calculation script may be strong. ClusterBid can help match capacity offers to the measured requirements once those requirements are explicit, but no marketplace listing can repair a benchmark whose workload and boundary were never defined.
Keep a one-page evidence request
Send providers the same request: metric definition; model and weights revision; precision and sparsity; accuracy or quality result; batch and concurrency; input and output distributions; warm-up and run duration; number of repetitions; complete hardware and topology; software versions; power settings; timer boundary; raw logs; and reproduction commands. Require the baseline to answer the same fields. A blank field should remain visible in the comparison sheet.
There is an honest limit to this checklist. Even a well-documented synthetic benchmark may not predict production behavior with changing traffic, failures, noisy neighbors, data pipelines, or evolving models. Treat the benchmark as a controlled measurement, then run a time-boxed workload trial with your own acceptance criteria. The purchasing decision should follow the trial's completed work, latency, quality, and failure observations, not the neatness of the vendor chart.