One counter cannot answer three questions
A quota is an entitlement boundary, not a utilization report and not a measure of completed work. Platform teams need at least three ledgers: capacity reserved or allocated to a team, capacity actually occupied over time, and successful work produced. If all three are called GPU usage, a team with idle reservations looks identical to a team running productive jobs, while failed jobs may look productive merely because they consumed hours.
Keep the units explicit. Entitlement is usually concurrent GPUs or a GPU-hour budget. Allocation is GPU count multiplied by scheduled wall time. Observed activity might be accelerator-active seconds, memory occupancy, or application-specific busy time. Successful work might be completed training steps, accepted images, or billable tokens. These measures are related, but none can be substituted for another without stating a policy choice.
Derive allocated capacity from scheduler time
The allocation ledger should begin when a GPU allocation becomes usable by the workload and stop when the allocation is released, not when the first kernel happens to run. This captures setup, data loading, checkpoint writing, and idle gaps that still block another tenant. Scheduler accounting systems expose job state and elapsed time for this purpose; Slurm's sacct is one example of a primary job-accounting interface, although its generic elapsed field must be joined to the recorded GPU count or GPU allocation record.
For a pod assigned 2 GPUs from 09:00 to 12:30, allocated capacity is 2 multiplied by 3.5 hours, or 7 GPU-hours. Suppose telemetry integrates each allocated device's active fraction over the interval and totals 4.2 hypothetical GPU-active-equivalent hours. Report both values. The 2.8 GPU-hour difference is an optimization signal, not evidence that the infrastructure was free. Another team could not necessarily use those fragmented idle moments.
Measure successful work outside the quota system
A scheduler can say that a job ran; it cannot decide whether the output was valid. Success belongs to the application: a training job may require a durable checkpoint and passing validation, while an inference batch may require responses accepted before their deadlines. Define one completion event, give it a stable workload ID, and join it to the allocation ledger without embedding product semantics in the quota controller.
Use a diagnostic sequence for apparent waste. First check whether allocated GPU-hours rose without a matching rise in successful units. Then split the gap into initialization, queue starvation, active computation, retry work, and post-processing. Finally inspect failures by reason. Low activity can be expected for communication-heavy or latency-sensitive work, so it is a diagnostic possibility rather than proof of poor engineering.
Build a four-column team chargeback
Consider an illustrative month with two teams and a hypothetical internal rate of $2.00 per allocated GPU-hour. Actual provider prices and internal rates vary. Team Atlas reserves 8 GPUs for 100 hours, occupies them for 720 GPU-hours, records 500 active-equivalent hours, and completes 90 accepted runs. Team Boreal reserves 4 GPUs for 100 hours, occupies them for 360 GPU-hours, records 300 active-equivalent hours, and completes 120 accepted runs.
Under allocation chargeback, Atlas owes 720 multiplied by $2.00, or $1,440, and Boreal owes 360 multiplied by $2.00, or $720. Showback then adds occupancy rate, active ratio, and successful output. Atlas used 720 divided by 800 reserved GPU-hours, or 90 percent, while Boreal used 360 divided by 400, also 90 percent. Their productivity differs, but finance should not silently convert that difference into a rate penalty.
| Team | Reserved GPU-hours | Allocated GPU-hours | Active-equivalent hours | Successful units | Illustrative charge |
|---|---|---|---|---|---|
| Atlas | 800 | 720 | 500 | 90 | $1,440 |
| Boreal | 400 | 360 | 300 | 120 | $720 |
Set policies from the right ledger
Use reservation data to enforce entitlement, allocation data to recover capacity cost, activity data to coach efficiency, and success data to evaluate product economics. A measurable reclamation rule could be: reduce a standing reservation at the next review when allocated use remains below 60 percent of reserved GPU-hours for four consecutive weeks and the team's documented launch buffer is not at risk. The threshold and window are illustrative governance choices.
For burst access, let teams exceed their baseline only from an explicitly uncommitted pool, and label those allocations preemptible. This preserves the meaning of a hard reservation. Avoid charging twice when a reserved pool is also assigned to a job: the invoice should state whether the commercial model bills the reservation, the allocation, or a base reservation plus an agreed burst premium.
Reconcile the ledgers before publishing
Run daily reconciliation across quota snapshots, scheduler assignments, telemetry identity, and application completion events. Flag orphan allocations, jobs with impossible negative or overlapping intervals, missing team labels, and success events without a workload record. The sum of team allocations should not exceed allocatable fleet GPU-hours for the same resource class and interval, except where documented virtual sharing changes the unit.
The limitation is attribution, not arithmetic. Shared inference workers, multiplexed GPUs, and distributed jobs make per-team activity estimates model-dependent. Publish allocation chargeback from auditable scheduler records, then present activity and successful-work ratios as operational evidence with known gaps. Keeping those fields separate makes disputes tractable and prevents a noisy utilization sample from rewriting an enforceable entitlement.