Why nightly evals disturb training even when GPUs look idle
Nightly evaluation often lands on training nodes because GPUs report low utilization between steps or during checkpoint writes. That reading misleads. Training throughput depends on steady step time, which needs predictable memory access, stable host to device feeding, and collectives that finish on schedule. An eval job that loads another weight copy, builds its own batches, and runs generation can disturb all three while utilization stays below one hundred percent.
The usual symptom is drift rather than failure. Step time grows by fractions of a second, curves arrive late, and the morning review suspects the network. Eval data loading can contend for host memory bandwidth and transfer slots while long eval prompts create uneven memory traffic. Training keeps running and simply waits longer each step, and those waits add up over thousands of steps.
Separation is a scheduling choice first. The aim is flat training step time by giving eval a bounded queue with its own concurrency cap. The cap matters more than the node count. An uncapped eval queue can still flood shared storage or shared network even after direct GPU sharing ends.
A separate queue with capped concurrency
Use two queues with different admission rules. Training keeps exclusive nodes and priority for collectives and checkpoint writes. Eval moves to its own queue with a fixed concurrency cap, memory request, and time limit. Set the cap from measured throughput. As an illustrative example, if one worker clears the suite in forty minutes inside a two hour window, a cap of two or three concurrent workers is a defensible start, tuned from logs.
Slurm expresses this pattern through partitions that group nodes into queues with distinct limits and shared or exclusive access, coordinated by controller and node daemons. Keep training exclusive with no preemption by eval. Give eval its own nodes or a time boxed reservation, a maximum concurrent count, and requeue on failure so one hung eval never pins training capacity.
| Placement | When it fits | Risk to step time | Operational cost |
|---|---|---|---|
| Share training GPUs | Smoke tests under five minutes | High, shared bandwidth and capacity | Low until drift appears |
| Same nodes, separate window | Owned cluster with idle gap | Medium, reload and storage overlap | Pause plus reload each night |
| Separate eval nodes | Nightly suites over thirty minutes | Low when storage is split | Extra nodes for peak hours |
| Hardware partition | Small fixed shape eval | Low compute risk, memory split stays | Less training memory during eval |
A workload derived rule for approving the split
Teams want one threshold. The brief mentions five percent added wall clock time as an illustrative team policy, not an established recommendation. Treat it as a starting example only. Five percent can be strict for a short fine tune and loose for a month long run where two percent still means days of delay. Replace it with a rule tied to your step logs and run length.
Measure a clean baseline first. Take at least two hundred steps after warmup with no eval present and exclude checkpoint steps. Repeat the same measurement on nights when eval shares resources. Approve a separate eval queue when the shared median exceeds the clean median by enough to shift the run plan, or when variance doubles for two nights in a row. Record the margin and the run length behind it.
Keep five fields per decision: clean median, shared median, steps per night, eval concurrency, and checkpoint minutes. Revisit the call when model size, sequence length, batch size, or suite size shifts by more than about ten percent. A rule tied to those inputs survives the next model. A borrowed constant does not.
Worked example from step time logs
Take an illustrative job where every number is a planning assumption, not a measurement from a real cluster. Clean training runs two point zero seconds per step over five hundred stable steps. Twelve thousand steps per night gives twenty four thousand seconds of step time, or six point six seven hours. Checkpoint writes add thirty minutes. Clean wall clock totals about seven point one seven hours.
With colocated eval over three nights, median step time rises to two point one two seconds. The rise is six percent, from two point one two minus two point zero divided by two point zero. Over twelve thousand steps that adds one thousand four hundred forty seconds, or twenty four minutes. Eval also adds ten minutes of storage contention during upload. Shared wall clock reaches about seven point seven three hours, near thirty four minutes longer. The illustrative five percent policy would trigger a split here, but the arithmetic is the durable point.
Compare node hours next. A separate two node eval pool that clears the suite in ninety minutes costs three node hours per night in illustrative hypothetical accounting. The colocated delay spreads about four point five extra GPU hours across an eight GPU job, from about three point two GPU hours of slower steps plus about one point three GPU hours of storage contention, plus late scores and noisy alerts. When measured eval hours exceed the onsite buffer, matching that measured need to rented capacity on ClusterBid avoids buying nodes for a ninety minute nightly peak. Actual rental prices vary and no price is quoted here, so use the time math as the main justification and treat all node hour figures as hypothetical planning assumptions.
Diagnostic sequence when step time stays noisy
When noise persists after the split, check placement first. Confirm with scheduler records that no eval job reached training nodes through backfill, shared access, or a stale node list. Slurm accounting and queue listings show where each job ran, which outweighs where the script intended it to run. One misplaced worker can explain a full night of drift.
Next look below the GPU. Compare host memory pressure, storage latency around checkpoints, and network retransmit counters between clean and noisy nights. Eval suites often fetch prompts and reference outputs from the same filesystem that receives checkpoints. If checkpoint minutes grow while step medians stay flat, suspect storage or network rather than HBM sharing.
Then inspect device signals before touching concurrency. Review memory used, bandwidth saturation, power draw, and clocks around eval start and stop. Saturated bandwidth with steady clocks makes memory contention a possibility, not proof. Falling clocks with rising heat points toward thermal or power limits. Change one variable per eval cycle and hold the cap steady until the logs identify the layer.
Limits of separation and when to keep sharing
Separation cannot fix every slowdown. It will not shorten checkpoint writes, repair slow storage, widen a tight fabric, or contain a suite that grows weekly. A separate pool can also sit idle most of the day on owned hardware. Where that idle cost dominates, a time separated window on the same nodes with training paused and weights unloaded may cost less, once pause and reload time are measured.
Medians also hide tails. A suite with shifting prompt lengths changes HBM pressure night to night, so a rule tuned once will drift. Version the eval suite with each checkpoint, log prompt length distribution with every result, and treat each split decision as valid only for the workload shape that produced it.