Record the decision boundary while it is still visible
A capacity decision log should answer what was decided, for which workloads, over what period, and by whom. Give each record a stable identifier, date, status, owner, and review date. State the decision in operational units such as reserved GPU-hours per month, simultaneous accelerators, node shape, region, and service tier. A note that the team chose more GPUs will not help when somebody later asks what more meant.
Treat the record as a close relative of an architecture decision record. The Architecture Decision Record project defines an ADR as a document that captures an important architectural decision together with its context and consequences. GPU capacity deserves similar treatment because contracts, data placement, queue policy, and application architecture can make a choice expensive to reverse.
Attach workload evidence, not meeting memory
Link a versioned evidence bundle: demand query, trace window, workload strata, benchmark manifest, raw results, quality gate, incident history, queue statistics, utilization telemetry, and approved commercial inputs. Record checksums or immutable object identifiers where possible. A dashboard link alone is fragile because its query and retention can change after the decision.
Define every observed metric. NVIDIA DCGM Exporter exposes GPU telemetry in Prometheus format and can identify fields such as GPU utilization and framebuffer memory used. Those measures describe aspects of device activity, not product value by themselves. Pair them with completed useful work, latency, failures, and queue delay. A high utilization chart cannot establish that the chosen capacity served the right tasks efficiently.
| Evidence item | Snapshot required | Question it supports |
|---|---|---|
| Demand trace | Dates, query, strata | How much work arrives? |
| Benchmark bundle | Artifact and load schedule | What can one configuration sustain? |
| Operations data | Queue, failures, maintenance | What headroom is consumed? |
| Commercial input | Quote ID and validity | What trade-off was available? |
Write assumptions as testable claims with owners
An assumption should have a value, unit, evidence source, owner, confidence, and expiry or trigger. Examples include monthly task growth, long-context share, retry factor, sustainable accepted tasks per GPU-hour, required peak headroom, delivery date, and effective billed utilization. Avoid words such as modest or ample when a range can be written.
Separate facts, forecasts, and constraints. Last month's measured peak is a fact about a defined window. Next quarter's peak is a forecast. A contract minimum or latency objective is a constraint. Mixing those categories makes later disagreement look like revisionism. It is legitimate for a forecast to be wrong; the record should make it possible to see which input was wrong and whether the decision rule was followed.
Compare real alternatives on one decision frame
Include the status quo and at least one meaningfully different option when they were available: remain on demand, reserve part of the baseline, procure dedicated nodes, reduce demand through software work, or accept queueing for flexible jobs. For each, show capacity unit, lead time, workload fit, operational burden, failure exposure, reversibility, and cost under the same demand cases. Mark an option unavailable if a hard constraint rules it out; do not quietly delete it.
Do not compress the comparison into one three-year total. Timing, utilization, and reversibility matter. A lower hypothetical unit price can lose if capacity sits idle or cannot run the workload. Conversely, an on-demand option can be rational despite a higher unit assumption when demand evidence is weak. Keep all dollar figures labeled hypothetical unless tied to a dated approved quote, and state that actual prices vary.
| Option | Strongest case | Main consequence | Reversal path |
|---|---|---|---|
| On-demand baseline | Uncertain or bursty demand | Variable availability and spend | Change placement policy |
| Partial reservation | Stable base plus bursts | Commitment management | Expire or modify at term |
| Dedicated capacity | Stable, specific workload | Operations and concentration risk | Resell, repurpose, or exit contract |
| Software reduction | Avoidable model work | Engineering time and delivery risk | Restore prior path |
Worked example: choose capacity without hiding uncertainty
Consider an illustrative six-month planning record. Measured base demand is 2,400 GPU-hours per month, the forecast range is 2,100 to 3,300, and peak concurrency is 12 GPUs. A tested optimization is expected to reduce execution demand by 10 percent, but has not shipped. Option A keeps all usage on demand at a hypothetical $3.00 per GPU-hour. At the base, that is 2,400 x $3.00 = $7,200 per month. Option B reserves an illustrative 1,800 GPU-hours at $2.20 and buys the remaining 600 at $3.00, or 1,800 x $2.20 + 600 x $3.00 = $5,760 per month.
The record chooses the partial reservation because 1,800 hours sit below the low forecast after a 10 percent reduction: 2,100 x 0.90 = 1,890. It keeps bursts flexible and avoids counting unshipped optimization twice. The hypothetical base saving is $7,200 - $5,760 = $1,440 per month. Actual prices vary, and the example omits storage, networking, taxes, support, and idle timing effects. The choice is contingent on the reserved capacity meeting the benchmarked workload and concurrency needs.
Set review and reversal triggers before commitment
A decision log becomes useful when it states what would cause a review. Use measurable triggers tied to the assumptions: observed demand below 1,890 GPU-hours for two consecutive months, forecast error above 15 percent at two closes, p95 queue delay above the service objective for seven days, a model change that moves accepted tasks per GPU-hour by 10 percent, or a provider configuration change that invalidates the acceptance test. Assign a person or role to each trigger.
At review, run a fixed sequence. Confirm the trigger and data window. Reproduce the original calculation with current inputs. Identify whether demand, workload mix, benchmark rate, price, or constraint changed. Re-score the original alternatives and any genuinely new option. Then retain, amend, or supersede the record. Do not overwrite the original rationale. Link the new record so the change in evidence remains visible.
Keep the log light enough to survive contact with operations
Use one page for the decision and links for evidence. Require a reviewer from the workload team and one from the team that owns the budget or service. Store the log where operators can search it, and index records by workload, provider, region, hardware family, status, and review date. ClusterBid can be referenced in the evidence when it helps compare available capacity against the measured requirement, but the decision record should remain portable across sourcing channels.
A log cannot remove uncertainty or guarantee that alternatives were fully known. Commercial terms can be confidential, benchmark evidence can age, and organizational constraints may change without clean telemetry. State those limitations and the confidence level. The payoff is not bureaucracy or perfect foresight. It is a durable explanation of why a reversible or irreversible capacity choice was reasonable with the evidence available on the recorded date.