Tune a pipeline, not a knob
Data loading is a pipeline of storage reads, decoding or transforms, collation, host buffering, host-to-device transfer, and GPU compute. Increasing worker count helps only while parallel CPU work or I/O concurrency is on the critical path. After that point, extra workers add scheduling overhead, can replicate parent-process Python objects that workers access, enlarge prefetched queues, or increase random access pressure without feeding the GPU faster. PyTorch 2.6 explicitly warns that accessed Python objects can make aggregate CPU memory approach worker count times the relevant parent-process footprint.
Pinned memory is similarly conditional. CUDA documentation explains that page-locked host buffers enable efficient asynchronous transfers and can support overlap, but also warns that excessive pinning can reduce system performance. The useful limit is therefore the smallest configuration that meets the throughput and tail-latency objective with adequate CPU and memory headroom.
Understand what the loader multiplies
In PyTorch, num_workers controls subprocesses used for loading. When workers are enabled, prefetch_factor is the number of batches loaded in advance by each worker. The PyTorch 2.6 documentation therefore describes eight workers and prefetch factor two as 16 batches prefetched across all workers. That documented count is not a complete memory bound: it excludes the batch being consumed, interprocess-transfer and pinning stages, worker dataset state, and framework overhead. Variable-size batches make byte usage less predictable than queue length.
Pinning adds another host-memory stage before transfer. If training uses non-blocking device copies and suitable streams, transfer can overlap other work, but the buffer cannot be safely reused until the copy completes. Persistent workers can avoid process creation between epochs while keeping worker state and memory alive. Measure steady state and epoch boundaries because one setting can improve one while worsening the other.
Build a controlled sweep
Fix model, batch construction, sample order, storage cache condition, CPU affinity policy, and GPU clocks or power policy where operationally appropriate. Warm up, then measure enough steps to stabilize p50 and p99. Start with workers 0, 1, 2, 4, and 8, stopping before physical CPU saturation. At each count, test pinning off and on. Test prefetch factors only after choosing a promising worker range.
Collect samples or tokens per second, batch-ready p50 and p99, GPU idle fraction, host-to-device duration, CPU utilization by core, run-queue length, storage throughput and latency, process and cgroup peak memory, locked or pinned bytes where observable, major faults, and swap activity. Repeat each point. A throughput number without host pressure and variance cannot identify a safe operating point.
Work through an illustrative result
Imagine an illustrative sweep with a fixed workload. Two workers with pinning deliver 920 samples per second, four deliver 1,050, eight deliver 1,070, and twelve deliver 1,055. Peak host memory is 42, 58, 91, and 128 GiB respectively. Moving from four to eight gains 20/1,050 = 1.9 percent while adding 33 GiB. Twelve workers regress while consuming another 37 GiB.
Choose four workers if the acceptance rule requires at least 5 percent throughput gain for each worker-count step and at least 25 percent node-memory headroom. The numbers are illustrative, not a benchmark, and actual results vary by data, transforms, CPU, accelerator, and storage. The knee can also move after changing batch size or sequence-length distribution.
| Workers | Illustrative throughput | Illustrative peak host memory | Decision |
|---|---|---|---|
| 2 | 920 samples/s | 42 GiB | Baseline candidate |
| 4 | 1,050 samples/s | 58 GiB | Useful gain |
| 8 | 1,070 samples/s | 91 GiB | Gain below 5% |
| 12 | 1,055 samples/s | 128 GiB | Regression |
Diagnose the shape of the curve
If throughput rises with workers while CPU idle remains available, keep sweeping. If CPU is saturated and run queues grow, optimize transforms, affinity, or core allocation before adding workers. If storage latency worsens with concurrency, reduce workers or reorganize data into larger sequential reads. If batch-ready time is low but GPU gaps remain, inspect transfer synchronization and model-side work rather than blaming the loader.
When pinning helps transfer time but hurts overall throughput, inspect locked-memory volume, allocation churn, and host pressure. NVIDIA advises measuring because pinning is heavyweight and scarce. When pinning changes little, CPU transforms or storage may dominate, or copies may not be arranged to overlap. Confirm with a timeline rather than assuming the flag created overlap.
Protect the chosen setting with guardrails
Select the lowest worker count within 2 percent of the best repeatable throughput, provided p99 batch-ready time meets the step budget, peak memory leaves the chosen reserve, and swap remains zero. The 2 percent tolerance is a policy example that favors operational margin. Record framework version, dataset layout, CPU allocation, batch size, prefetch factor, pinning, persistence, and measurement window beside the result.
Retest after material changes to transforms, dataset encoding, storage location, batch shape, CPU quota, or accelerator count. A limitation is that cache state and shared-cluster contention can dominate a short sweep, so test both cold and warm conditions and report variance. If the measured pipeline needs a different CPU-to-GPU ratio or more host memory, choose infrastructure from that evidence rather than a universal workers-per-GPU rule.
Guard against worker failure and drift
Add a failure guard to the benchmark. Terminate one worker or inject a corrupt sample and confirm that the loader reports the problem promptly rather than leaving the training loop waiting. Record worker restart behavior and any queue memory retained after failure. Throughput tuning should not turn a visible data error into an indefinite distributed stall during a long multi-node training run.
For production, watch the chosen point for drift. Alert when batch-ready p99 exceeds its accepted value for several windows, peak host memory approaches the reserve, swap becomes nonzero, or throughput falls while GPU compute time is stable. Rerun the sweep instead of incrementing workers automatically. The former optimum can become unsafe when records grow, transformations change, or multiple jobs begin sharing the same CPU and storage path. Preserve the last accepted configuration so operators can roll back a tuning change without reconstructing the benchmark. Run a short canary with the proposed configuration before applying it to long jobs, and compare both throughput and memory against the stored baseline.