Separate reservation, use, and service obligation
A shared GPU pool exists for more than the work visible in utilization charts. Research creates variable jobs and can often queue. Production may require immediate capacity, redundancy, predictable latency, and an on-call response even when traffic is quiet. A fair model therefore separates three causes of cost: capacity reserved for a team, resources measured while its jobs run, and service objectives that force the operator to hold headroom or provide extra support.
This separation avoids two common distortions. Charging only by average GPU utilization makes the production team look cheap when its standby capacity is idle, although its availability requirement caused that capacity to exist. Charging only by reservation can make researchers pay for blocked time they neither requested nor controlled. Cost allocation should explain why capacity was acquired and held, as well as which workloads consumed it.
Define the ledger before choosing rates
Build a monthly ledger with capacity cost, facilities or hosting, network and storage, platform software, operations, and any directly attributable support. Keep financial treatment aligned with the organization's accounting policy; the operational model is not a substitute for accounting advice. For each item, label it direct, shared by measured activity, shared by reservation, or centrally funded. FinOps guidance explicitly recognizes multiple shared-cost methods, including fixed, proportional, proxy-based, and central-budget treatment.
Attach ownership metadata at job submission: team, environment, project, cost center, and service class. Scheduler accounting can supply allocated GPU time, CPU, and memory. Energy, GPU memory, and GPU utilization require the relevant accounting or profiling plugins and supported hardware, so do not assume every Slurm installation records them. Slurm's TRES model can track configured GPUs and other resources and can calculate weighted billing units. Those units are an input to an internal allocation method, not proof that one weight is economically correct.
Charge fixed capacity to the demand driver
Allocate a reserved slice to the team that requires exclusivity or an immediate-start objective. If production reserves eight GPUs continuously, its reservation basis is 8 GPUs multiplied by the hours in the month, regardless of kernel occupancy. If research has no reserved minimum and accepts queueing, it should not inherit that reservation charge merely because it fills idle slots opportunistically.
For jointly protected headroom, use an agreed demand driver. Examples include each service's guaranteed GPU count, peak admitted workload, or contribution to a documented recovery target. Do not distribute headroom by observed utilization alone. Idle reserve is doing work by preserving an option. If leadership decides that emergency capacity benefits the whole company, show it as a central platform line rather than hiding it in a per-job rate.
Measure consumption with allocations, then inspect efficiency
Use allocated GPU-hours as the primary job-level quantity because allocation is stable, auditable, and linked to scheduling decisions. A four-GPU job holding its allocation for three hours consumes 12 allocated GPU-hours even if device utilization oscillates. Add memory, CPU, storage, or network weights only when they materially drive pool cost or constrain access. Publish the weights and review them on a fixed cadence.
Utilization remains valuable as an engineering diagnostic. Low accelerator activity can reveal input stalls, oversized requests, synchronization, or an allocation that should shrink. It should not automatically erase cost responsibility, since another team could not use an exclusively allocated device. Report both allocated GPU-hours and a workload output such as trained tokens, completed experiments, or served requests. That pairing separates accountability from optimization.
Worked example: allocate one illustrative month
Assume an illustrative monthly pool cost of $120,000; actual prices and internal costs vary. Policy assigns $72,000 to fixed capacity and $48,000 to variable operation. Production reserves 60 percent of capacity and research reserves 20 percent, leaving 20 percent as centrally funded resilience and scheduling slack. Reservation charges are $72,000 times 0.60, or $43,200, to production; $72,000 times 0.20, or $14,400, to research; and $14,400 to the platform budget.
During the month, measured eligible use is 18,000 allocated GPU-hours for production and 12,000 for research. The variable rate is $48,000 divided by 30,000, or a hypothetical $1.60 per allocated GPU-hour. Production receives $28,800 of variable cost and totals $72,000. Research receives $19,200 and totals $33,600. The platform retains $14,400. The totals reconcile to $120,000, and no team is paid a credit for using headroom that another service required. This example assumes the $48,000 variable ledger contains only costs intended for usage allocation, so it does not charge the same fixed capacity twice.
Make service levels visible in the allocation
Define service classes in operational terms: queue-start percentile, preemption policy, maintenance notice, recovery target, support coverage, and whether capacity is dedicated or fungible. Then identify which incremental costs each class creates. A production class might fund reserved GPUs and after-hours operations. A research class might accept preemption and maintenance windows in exchange for avoiding reservation charges.
Use a measurable rule for changing the model. For example, review a reservation when its rolling eight-week peak allocated demand stays below 60 percent of reserved capacity and its start-time objective would still be met after reducing one GPU unit. This is an illustrative governance threshold, not a universal optimum. Changes should take effect prospectively, because retroactive weights turn cost reporting into negotiation rather than a predictable control system.
Handle preemption, retries, and failed work explicitly
When research backfills production reserve and is preempted, decide in advance whether the interrupted GPU-hours are charged to research, the production reserve, or a platform interruption pool. A sensible policy charges successful opportunistic consumption to research but assigns provider-caused or policy-caused rework separately, provided the scheduler can identify it. Otherwise teams may avoid useful backfill because interruption makes its apparent unit cost unpredictable.
Failed jobs need reason codes. User configuration errors, data failures, infrastructure faults, and planned preemption should not be blended. Allocate the immediate resource consumption according to the published rule, then report avoidable rework as a separate metric. The purpose is not punishment. It is to preserve a signal that shows where reliability work or job validation could reduce total demand.
Reconcile totals and publish the model's limits
Each period, require four checks: all ledger cost is allocated or explicitly central; all eligible jobs have valid ownership; measured units reconcile with scheduler records; and reservation assignments match approved service objectives. Flag unowned usage rather than spreading it silently. A practical decision rule is to block chargeback publication if more than an illustrative 2 percent of pool cost or GPU-hours remains unexplained, while still issuing a provisional showback report.
No allocation model is perfectly causal. Shared networking, staff time, capacity fragmentation, multi-GPU topology, and future demand do not reduce to one meter. More granular attribution also costs engineering time and may create false precision. Start with reservation, allocated use, and service class, then add dimensions only when they change a decision. The best model is the simplest one stakeholders can audit and operators can sustain.