Short must be a scheduling contract
A short-job queue works only when short has an enforceable meaning. Define it with a wall-time ceiling, GPU-count ceiling, eligible hardware classes, preemption policy, and consequence for overruns. A label supplied by the submitter is not enough. Without boundaries, long jobs migrate into the fast lane and restore the wait time the lane was meant to remove.
Start with two or three classes, not a taxonomy for every team. For example, an interactive experiment class might allow one to four GPUs for at most 30 minutes, a short class one to eight GPUs for at most four hours, and the regular queue everything else. These limits are illustrative. Derive production limits from observed runtime and demand distributions, then publish them as policy rather than folklore.
Runtime estimates enable safe backfill
A backfill scheduler can start a lower-priority job when doing so does not delay the expected start of higher-priority work. That decision depends on requested time limits and all requested resources. An inflated limit makes a short experiment look too large for available holes; an optimistic limit risks termination at the configured time limit before useful results are written. Slurm's scheduling documentation explicitly connects backfill planning with reasonably accurate job time limits.
Store requested and actual runtime for every job class. Report the ratio and absolute error, then coach or automatically adjust chronic over-requesters. For repeatable experiments, a launcher can suggest a limit from prior runs of the same workload signature while leaving safety margin for variance. The estimate should include setup, data staging, evaluation, and checkpoint flushing, not only the training loop. Track cold and warm launches separately when environment or dataset caches make their runtime distributions materially different.
Fragmentation needs both space and time measures
Free GPU count is not sufficient. Eight idle GPUs spread one per node cannot satisfy an eight-GPU single-node request, while a full node reserved to start later may contain a temporal hole that a 20-minute job can safely use. Measure spatial fragmentation as idle GPUs that cannot satisfy queued shapes, and temporal fragmentation as reservation gaps left unused despite eligible short jobs.
Break the metrics down by GPU class and node shape. Track stranded GPU-minutes, backfill opportunity capture, and the top blocked request dimensions. If most stranded capacity comes from four-GPU jobs leaving four GPUs unusable on eight-GPU nodes, consider packing compatible jobs, a four-GPU class, or whole-node allocation. Do not infer the remedy from utilization alone; topology, isolation, and memory capacity may make the apparent gap unusable.
A worked window shows the admission decision
Consider an illustrative eight-GPU node that has a reservation for all eight GPUs in 70 minutes. Job A asks for four GPUs and 30 minutes; Job B asks for eight GPUs and 90 minutes; Job C asks for two GPUs and 60 minutes. A and C can fit concurrently by GPU count, and both claim completion before the reservation. B cannot fit the temporal window. The scheduler should still account for launch overhead and a safety margin.
If policy reserves 10 minutes for cleanup, the usable window is 70 minus 10 = 60 minutes. A fits because 30 is at most 60. C fits exactly at 60, but an operator may reject exact-boundary fits if runtime estimates have poor calibration. B fails because 90 exceeds 60. These values are illustrative. Actual scheduling also depends on priority, topology, licenses, memory, and other resources.
| Illustrative job | GPU request | Time request | Fits 60-minute window |
|---|---|---|---|
| A | 4 | 30 min | Yes |
| B | 8 | 90 min | No |
| C | 2 | 60 min | Policy-dependent boundary |
Use a diagnostic sequence for queue complaints
When a short job waits, first confirm it is eligible for the class and requested an available GPU type. Second inspect its reason code and higher-priority reservations. Third test whether its GPU and node shape fits current holes. Fourth compare requested time with the available backfill window. Fifth inspect fair-share, age, and class factors. Sixth look for dependency, data, license, or prolog constraints. This prevents every delay from being mislabeled as insufficient capacity.
When short jobs start but fail to finish, compare actual runtime with requests, termination reason, startup overhead, and checkpoint behavior. A queue can have excellent start latency while producing little science if jobs spend most of their allocation importing environments or staging data. Measure successful experiment completions per allocated GPU-hour alongside queue latency.
Operate the queue to explicit objectives
A practical scorecard includes p50 and p95 eligible wait time, starvation count, runtime-estimate error, percentage of short jobs that finish within limit, stranded GPU-minutes, and the share of backfill windows captured. Set an error budget, such as at least 95 percent of eligible short jobs starting within twice their requested duration during business hours. The target is illustrative and should be tested against regular-job delay.
No queue policy creates capacity. During sustained saturation, faster short-job starts transfer delay elsewhere unless quotas, preemption, or reserved capacity define who pays. Review the trade monthly with per-team consumption and wait distributions. Consider additional capacity only after separating true saturation from poor estimates, impossible request shapes, and policy-induced fragmentation patterns.