Locality is a chain, not a CPU-only setting
A GPU input path can involve storage, page cache, data-loader workers, host memory, a PCIe root complex, and the GPU. A distributed path adds a NIC and possibly GPU Direct RDMA. NUMA placement affects where CPU threads run and where their pages are allocated, but it does not collapse every hop into one locality setting. GPU and NIC paths still depend on the machine's PCI topology.
Linux describes NUMA as non-uniform access among memory nodes and lets administrators restrict CPU affinity or apply memory policy. It also warns that task migration and page placement have distinct controls. Pinning a process to CPUs near a GPU while its buffers remain on another memory node can preserve remote traffic. The reverse can happen when memory is local but interrupt or loader work runs elsewhere.
Map four resources before changing affinity
For every training rank, record its CPU set, allowed memory nodes, GPU PCI address, and selected NIC PCI address. Linux PCI sysfs exposes a read-only local_cpus mask described as the nearby CPU mask, but that hint does not by itself prove the complete device path. System topology tools can show sockets, NUMA nodes, bridges, and devices, and NCCL also relies on sysfs to discover GPU and network-card PCI topology. Preserve the map from inside the same container or job namespace used by the workload.
Do not assume device numbering communicates locality. GPU 0 and NIC 0 may be names assigned by enumeration rather than a physical pair. Likewise, two GPUs on one socket may have different upstream bridges. Build an explicit table keyed by PCI address, then annotate each workload rank. If topology is virtualized or sysfs is incomplete, mark the map as uncertain and rely more heavily on measured comparisons.
| Resource | Evidence to capture | Common mistaken assumption |
|---|---|---|
| CPU | Allowed CPU list and socket or NUMA node | The scheduler will keep workers local |
| Memory | Policy and observed page placement | CPU pinning moves existing pages |
| GPU | PCI address and upstream path | GPU index equals socket index |
| NIC | Selected interface, PCI address, and port | The first interface is nearest every GPU |
Account for allocation time and first touch
The Linux kernel's default policy generally allocates a page from memory local to the CPU that faults it, while explicit policies can bind, prefer, or interleave allocations. A task policy governs subsequent allocations in its scope; changing policy alone does not migrate pages already faulted elsewhere. That makes thread placement and startup order part of the experiment.
Set affinity and memory policy before creating loader workers and before allocating or touching large host buffers. Check the actual process tree because child workers may inherit or alter affinity. Shared memory and file-backed page cache can follow rules that differ from anonymous buffers. A policy visible on the parent process is therefore evidence about intent, not proof of every page's location.
Measure the unpinned baseline before enforcing a theory
Run a stable workload segment with the default scheduler and memory behavior. Capture samples per second, GPU idle gaps attributable to input waiting if observable, host-to-device transfer time, CPU utilization by node, memory bandwidth, remote NUMA accesses where supported, and network collective time. Keep storage cache state, dataset shard, worker count, prefetch depth, batch shape, and augmentation fixed.
Then test one intervention at a time: CPU affinity only, CPU plus memory policy, and CPU plus memory plus NIC-aware rank placement. Recreate workers and buffers between cases so old page placement does not contaminate the comparison. Randomize or alternate case order if thermal behavior or page cache warming could trend over time. Collect at least one repeat after returning to the baseline, since an apparently faster pinned case may simply inherit a warmer cache.
Work through a two-socket placement
Consider an illustrative two-socket node with four GPUs and one NIC local to each socket. Rank 0 uses a GPU and NIC under socket 0, but eight loader workers run across both sockets. Candidate A leaves workers unpinned. Candidate B gives rank 0 four workers on socket 0 and allocates their new buffers from node 0. Candidate C pins workers but interleaves memory across nodes. These are test cases, not claims about the best layout.
Suppose illustrative steady-state rates are 1,000, 1,080, and 1,030 samples per second. Candidate B improves on A by (1,080 - 1,000) / 1,000 x 100 = 8 percent. If its p95 input-wait interval also falls from 42 ms to 28 ms, the result supports keeping the policy for this workload. Actual gains vary, and a change in worker count, caching, or augmentation can reverse the outcome.
Retain pinning only when the gain survives repetition
Use a predefined rule. An illustrative rule retains an affinity policy only when three repeated runs improve median useful throughput by at least 3 percent, do not worsen p95 step time by more than 2 percent, and do not create memory pressure or starve network progress threads. These thresholds are local operating choices, not NUMA constants.
If throughput improves but network time worsens, the CPU set may be too narrow or may place communication threads poorly. If host-to-device copies improve but end-to-end steps do not, the input path was not limiting. If only the first run improves, cache state or page placement may be the real differentiator. Report both the intervention and the full workload outcome.
Know what NUMA tuning cannot solve
Affinity cannot fix slow storage, undersized host memory, decompression that exceeds available cores, a weak PCIe path, or a congested network. Strict binding can also reduce resilience when a local node is under memory pressure. Containers and job schedulers may restrict CPU and memory sets, and Linux memory policy operates within those cpuset constraints.
Document the workload envelope for any retained policy, including node model, BIOS NUMA mode, kernel, driver, container limits, GPU and NIC mapping, worker count, and data shape. Revalidate after hardware or scheduler changes. Include a safe fallback that removes the custom binding if the expected CPUs or memory nodes are absent. When comparing capacity through ClusterBid, request topology evidence that supports the tested placement, then repeat the benchmark on the delivered system rather than assuming the same device indices imply the same paths.