Sizing a separate GPU pool for continuous safety evaluations

Safety suites behave differently from training jobs. They arrive on every model checkpoint, every data change, and every release candidate, then they need a fast and repeatable answer. When those jobs borrow training nodes, they wait behind long running distributed jobs, compete for the same network fabric, and inherit whatever driver or library state the training job left behind.

Training schedulers also optimize for throughput of large jobs, not for latency of small frequent jobs. A small safety suite can sit queued while the scheduler packs a large training job, even when total utilization looks moderate. That delay turns safety checks into a release bottleneck, which creates pressure to skip them.

Isolation solves the scheduling conflict rather than the raw capacity shortage. A small pool reserved for safety work gives each suite a known start time, a clean software environment, and a stable hardware profile. Safety suites are usually short and bursty, but the pool must be sized from measured demand rather than from leftover capacity.

Start with two measured quantities for each safety suite: wall clock suite duration on a fixed GPU type, and arrival rate per day. Duration must be measured end to end, including model load, dataset read, generation, and scoring, not only the generation loop. Arrival rate must count scheduled runs plus reruns triggered by failures and release candidates.

Record duration separately for each GPU type in the pool. Record arrivals by trigger type as well, since nightly regression runs are predictable while per commit checks are bursty. Two weeks of logs is usually enough to see the daily pattern and the retry rate.

Do not use vendor peak throughput numbers for this step. Use the median suite duration and the 90th percentile duration from your own logs. The median sizes the common case and the 90th percentile protects release days when inputs are larger and the model is slower to load.

Convert demand into GPU hours per day by multiplying suite duration by arrivals. If a safety suite uses 2 GPUs for 0.75 hours and runs 8 times per day, demand is 12 GPU hours per day. Divide by usable hours per GPU per day to get the baseline GPU count. Usable hours should subtract time for health checks and image pulls, so use a measured overhead allowance; 20 usable hours from a 24 hour day is used here only as an illustrative assumption, not a default for all clusters.

Add a margin for burst overlap rather than assuming arrivals spread evenly. A simple workload derived rule is to provision for the busiest observed 4 hour window, not the daily average. If 5 of those 8 runs land between 14:00 and 18:00, the pool must handle 7.5 GPU hours inside 4 clock hours, which needs at least 2 GPUs before any queueing margin.

A separate pool also needs one spare slot for failure handling. Safety work often blocks releases, so a single GPU failure should not stop all suites. For pools of 2 to 4 GPUs, one additional GPU or an agreement to borrow one training GPU during maintenance is usually enough.

Isolation must be enforced by the scheduler, not by convention. In Kubernetes, that means a dedicated node pool with a taint that only safety evaluation Jobs tolerate, plus resource requests that prevent oversubscription. The Kubernetes documentation on taints and tolerations describes this mechanism directly, and the documentation on Jobs describes how to run finite batch work to completion with retries and parallelism. This scheduler and environment separation does not imply physical separation, since nodes can still share power, network fabric, or storage backends.

Keep the safety pool on a fixed container image, fixed model cache layout, and fixed GPU driver version. Training nodes change frequently as experiments update libraries. Pin the evaluation image digest and update it on a schedule with a recorded validation run.

Separate storage paths as well. Safety logs, model artifacts, and datasets should live under their own prefixes with their own quotas. Shared scratch directories create a failure mode where a large training checkpoint fills the disk and the next safety run fails during model download.

Consider an illustrative team with one safety suite that needs 2 GPUs for 45 minutes per run. Logs show 6 scheduled runs per day plus an average of 2 reruns, so 8 runs per day. Daily demand is 8 runs times 1.5 GPU hours per run, which equals 12 GPU hours per day. All numbers in this example are illustrative planning inputs, not measured industry values.

The busiest 4 hour window holds 5 runs, or 7.5 GPU hours. Two GPUs provide 8 GPU hours inside that window, which covers the burst only if nothing fails and no run overruns. The team provisions 3 GPUs of the same type: 2 for steady burst service and 1 for overlap, retry, and maintenance coverage. Usable capacity is estimated at 20 hours per GPU per day after health checks and image pulls, or 60 GPU hours total, against 12 GPU hours of demand.

Queue wait stays under 5 minutes on ordinary days in this setup. On release days arrivals rise to 12 runs, or 18 GPU hours, and the same 3 GPUs still clear the backlog within the working day. If arrivals rise permanently to 14 runs per day, the burst window math no longer closes, and the next section gives the rule for adding capacity. Where matching rented capacity to this measured requirement helps, teams sometimes compare options on ClusterBid before committing to a larger reservation.

Track one metric daily: the longest queue wait for a safety suite, measured from job submission to first pod running. Also track the count of days per week when any suite waits longer than the team target. The brief for this article suggests adding capacity when queue wait exceeds 15 minutes on three consecutive days. Treat that 15 minute and three day combination as an illustrative team policy, not an established industry recommendation.

Replace or confirm that policy with a workload derived rule before enforcing it. A practical derived rule is to add one GPU when the 90th percentile queue wait over a rolling 7 days exceeds 25 percent of median suite duration, or when burst window demand exceeds 80 percent of usable burst capacity for two consecutive weeks. Both versions tie the trigger to measured suite behavior rather than to a fixed minute value.

When the trigger fires, add one GPU of the same type rather than redesigning the pool. Mixed GPU types complicate duration statistics and queue predictions. Revisit the pool size quarterly, or after any change that alters suite duration by more than 20 percent.

A small fixed pool cannot absorb every spike. Release weeks, incident reruns, and dataset refreshes can double arrivals for a few days. The pool design should state explicitly which traffic gets priority during those spikes: release blocking suites first, then nightly regression, then experimental checks. Without that order, the pool fills with low value reruns while the release gate waits.

Hardware uniformity is another limit. A pool sized around one GPU type gives clean duration statistics, but a single hardware failure removes a large share of capacity. Monitor GPU error counters and completed versus failed suite counts per node.

Finally, isolation does not remove the need for cost review. A pool at 20 percent average utilization is still justified when it protects release cadence, but the team should know the cost per evaluated checkpoint. Divide monthly pool cost by completed safety runs to get that figure. If hypothetical planning is needed, use an explicitly hypothetical rate such as 2.50 dollars per GPU hour, show the arithmetic, and state that actual prices vary by provider, region, and commitment.

Kubernetes: Taints and Tolerations

Kubernetes: Jobs

GPU Inference Rolling Updates Without Capacity Surprises

Cable and Transceiver Changes Deserve a GPU Cluster Retest