Hierarchical Quota Models
Most GPU clusters running 256+ GPUs adopt a hierarchical quota model where the organization allocates GPU-time budgets to divisions, divisions to teams, and teams to projects. This mirrors the cost center structure that finance departments use for chargeback. Each level in the hierarchy has a guaranteed minimum (reserved) and a maximum ceiling (burst limit) expressed in GPU-hours per week or month.
Two implementation approaches dominate. Kubernetes-based clusters use ResourceQuota and LimitRange objects with custom schedulers that track GPU-time consumption against budgets. Slurm-based clusters use FairShare tree accounting with the sacctmgr tool to define associations and enforce limits. The Kubernetes approach offers finer granularity, while Slurm provides better predictability for tightly-coupled distributed training jobs.
Priority Scheduling and Preemption
Production clusters assign at least four priority tiers: critical (production inference serving, 10% of cluster), high (training runs with near-term deadlines, 20%), normal (standard training experiments, 50%), and low (exploratory research, spot-like, 20%). Jobs at higher tiers can preempt jobs at lower tiers, with the preempted job receiving a checkpoint save signal before GPU resources are reclaimed.
The preemption policy must balance fairness against utilization. A common approach is to allow preemption only when the lower-tier job has been running for more than 30 minutes, preventing wasteful repeated preemption of short jobs. Preempted jobs receive priority requeuing at the head of their tier queue, and the scheduler tracks a preemption credit score to prevent any single job from being preempted more than three times in a 24-hour window.
| Tier | Cluster Share | Preemption Behavior | Use Case |
|---|---|---|---|
| Critical | 10% | Preempts all lower tiers | Production inference, API serving |
| High | 20% | Preempts Normal and Low | Deadline-driven training runs |
| Normal | 50% | Preempts Low only | Standard experiments, fine-tuning |
| Low | 20% | Never preempts others | Hyperparameter search, research |
Cost Attribution and Chargeback
Cost attribution in shared GPU clusters requires tracking GPU-time consumed per project, per user, and per job, then mapping that usage to the cluster's blended cost per GPU-hour. The blended rate includes the GPU rental or depreciation, power (typically $0.10-$0.15 per kWh), network fabric amortization, storage, and datacenter overhead. For an H100 cluster, the fully-loaded blended rate ranges from $2.50 to $4.00 per GPU-hour depending on utilization.
Effective chargeback reports show cost at three levels: aggregate (total team spend), by job type (training vs inference vs data preprocessing), and by efficiency (GPU utilization percentage per job). Teams that consistently achieve above 70% GPU utilization effectively subsidize teams running at 30% utilization. The best governance systems apply a utilization bonus, charging above-70% teams 15% less per GPU-hour.
Utilization Gates and Idle Time Policies
The single largest waste in multi-tenant GPU clusters is idle time from jobs that acquire GPUs but underutilize them. A 256-GPU cluster at 50% average utilization wastes $1.5M to $2.5M per year in compute that was allocated but not used. Automated idle detection policies help: GPUs with below 30% utilization for more than 10 minutes receive a warning; after 30 minutes of idle, the job is suspended and its state checkpointed.
Practical implementation uses tools like DCGM (NVIDIA Data Center GPU Manager) for real-time utilization monitoring, with Prometheus recording per-GPU metrics and a Kubernetes mutating webhook that injects sidecar containers to report usage. Jobs that hit idle limits three times are flagged for manual review and their priority tier is automatically demoted by one level.
Fairness Algorithms: Dominant Resource Fairness and Beyond
Dominant Resource Fairness (DRF) is the standard fairness model for multi-tenant GPU clusters. It ensures that each tenant gets a share proportional to their allocation across the dominant resource dimension, which might be GPU count for training workloads or GPU memory for inference workloads. However, DRF breaks down when teams have heterogeneous job profiles requiring different ratios of GPU, memory, and network bandwidth.
A better model for AI clusters is weighted DRF with decay: each team receives a weight based on their funding contribution to the cluster, and unused quota decays with a half-life of 7 days. This prevents the tragedy of the commons where teams reserve maximum quota perpetually even when not running jobs. Under this model, teams that reserve 64 GPUs but use 8 retain only 16 GPU equivalents of priority after one week without usage.
Audit Trails and Compliance
GPU governance for regulated industries (healthcare, finance, defense) requires immutable audit trails covering every GPU allocation event. Each job record must capture user identity, project code, GPU count and type, start and end timestamps, and a reference to the approved budget allocation. This data feeds both cost reports and compliance audits for HIPAA, SOC 2, or internal controls.
Leading clusters implement a governance service mesh where every GPU allocation request passes through an admission controller that validates budget availability, quota limits, and data locality requirements before the job is scheduled. The controller writes a signed event to an append-only log. At 256 GPUs with 50-150 jobs per day, the audit system processes roughly 500,000 allocation events per month with sub-millisecond overhead per event.