Assign every byte a lifetime
Local NVMe can remove a shared-storage bottleneck, but node-local scratch should be treated as proximity, not durability. Kubernetes documents that emptyDir data is deleted when a Pod is removed from a node and that a local PersistentVolume remains tied to one node through node affinity. Neither mechanism by itself provides a second durable copy that survives loss of that node. The first design question is therefore categorical: can this file be reconstructed from a durable source within the recovery objective? If the answer is no, it does not belong only on scratch.
Classify paths before tuning throughput. Read-through dataset chunks and compiled artifacts are caches. Shuffle runs and temporary preprocessing output are intermediate state. A checkpoint is durable only after the complete object has been committed to durable storage and verified. Metrics, final model artifacts, manifests, and irreplaceable generated data are outputs. The directory name does not determine the class. A path called checkpoints on a local disk is still disposable.
Separate cache, shuffle, checkpoint, and output traffic
Different file classes create different I/O patterns and failure consequences. Dataset caching is usually read-heavy and can be evicted by a least-recently-used policy. Shuffle and sort spills can be write-heavy and need headroom until a stage boundary. Checkpoint staging produces bursts and must not report success until upload completes. Durable outputs should bypass local-only retention or use an explicit two-phase publication flow.
Use separate directories, quotas, and metrics even if all paths share one device. This prevents a cache fill from consuming the space reserved for a checkpoint staging file. Track bytes used and free, inode availability, read and write latency, eviction volume, durable-upload backlog, and oldest uncommitted artifact age. Kubernetes can track and limit ephemeral storage only under supported filesystem layouts, so verify what the node agent actually observes rather than assuming every mounted disk is accounted for.
| File class | Local policy | Recovery source | Completion signal |
|---|---|---|---|
| Dataset cache | Evictable | Object or shared storage | Checksum verified |
| Shuffle data | Stage scoped | Recompute stage | Consumer finished |
| Checkpoint staging | Protected until upload | Training process | Durable manifest committed |
| Final output | Transit only | None until published | Durable copy verified |
Build a reconstructable cache
Cache entries need content identity, not only human-readable filenames. Store the durable object version or checksum beside each entry, write downloads to a temporary name, verify length and digest, then rename into the visible cache. On restart, incomplete temporary files can be discarded. If multiple workers populate the same path, use an atomic claim or tolerate duplicate downloads without exposing partial content.
Treat cache warming as part of job startup and measure it. A large cache can make steady-state training fast while making replacement nodes spend hours idle. Report cold-start time, warm-hit rate, bytes fetched per training hour, and training throughput after warm-up separately. The limitation is that a synthetic sequential-read test will not predict a real pipeline that mixes decompression, random access, metadata lookup, and CPU transforms.
Make checkpoint publication transactional
A safe checkpoint flow has at least three states: writing locally, uploading durable objects, and committed. Write rank shards into a generation-specific staging directory. Close and synchronize files according to the application’s durability needs, upload them under a unique destination prefix, verify expected sizes or hashes, and publish the manifest last. Readers should only discover generations with a committed manifest. Overwriting a single mutable pointer invites a partially mixed checkpoint after interruption.
Do not delete the previous durable generation merely because the next local write finished. Retain it until the new durable manifest passes a restore check. If asynchronous upload is used, expose the difference between checkpoint creation time and durable commit time. A job that says saved at step 20,000 while 200 GB remains only on the node has not met a recovery objective tied to durable storage.
Reserve capacity from the peak, not the average
Consider an illustrative node with 3.2 TB of usable scratch, using decimal units where 1 TB equals 1,000 GB. Reserve 15 percent for operational headroom, leaving 3.2 x 0.85 = 2.72 TB. Suppose the protected checkpoint staging budget is two generations at 420 GB each, or 840 GB and therefore 0.84 TB, and shuffle may peak at 600 GB, or 0.60 TB. The remaining cache ceiling is 2.72 TB minus 0.84 TB minus 0.60 TB = 1.28 TB. The numbers are illustrative, and actual formatted capacity and workload sizes vary.
Turn the arithmetic into admission control. Reject a stage before it starts if estimated peak cache plus shuffle plus protected checkpoint bytes would exceed the usable budget. Begin cache eviction at a lower watermark, such as 75 percent used, and stop new local checkpoint creation at a higher watermark unless enough protected space is already reserved. Those thresholds are illustrative. The important property is that checkpoint headroom cannot be consumed by opportunistic cache data.
Rehearse the node disappearing
A recovery drill should terminate a worker while it has cached input, live shuffle files, and a checkpoint upload in progress. Reschedule onto a clean node. Confirm that partial cache entries are rejected, shuffle work is recomputed or resumed from a documented boundary, the incomplete checkpoint is ignored, and the last committed durable checkpoint loads. Measure time to useful training, not merely time until the replacement process starts.
Set a measurable rule such as: after any single-node loss, no acknowledged durable output is missing, the job resumes from the newest committed manifest, and training returns to 90 percent of pre-failure throughput within the declared recovery target. The 90 percent figure is illustrative. If the dataset origin cannot sustain simultaneous cache refills across many nodes, a one-node drill understates correlated recovery risk and a staged multi-node exercise is needed.
Operate scratch as a budget
Alert on trends before the filesystem is full: projected time to the high watermark, upload queue age, eviction rate, inode exhaustion, and write-latency percentiles. Kubernetes can evict Pods in response to node filesystem pressure, while a local PersistentVolume is accessible only through its bound node. Job-level monitoring should therefore connect storage pressure and node availability to checkpoint freshness and recoverability, not display free bytes in isolation.
Choose capacity from measured working sets and failure recovery, then match nodes to those requirements. ClusterBid can help match GPU capacity only after the request includes scratch bytes per node, cold-fill bandwidth, durable-storage path, and acceptable rebuild time. More local NVMe does not repair an undefined durability boundary. The honest limitation is that redundant local devices may improve device-failure tolerance but do not protect against node loss, operator deletion, or a failed upload protocol.