Cold start is a chain, not one timer
A replica does not become useful when its container process exists. It becomes useful when the correct image is present, the requested model revision is available, the artifacts have passed required checks, weights occupy their runtime locations, and the serving endpoint is ready. Treat those as five spans: image pull, artifact fetch, verification, load, and readiness. A single pod-start duration hides which span changed and encourages the wrong fix.
Instrument each boundary from an external controller as well as inside the process. Record image digest, model revision, node identity, cache state, bytes expected, bytes read remotely, and final readiness time. Internal logs can explain loading, but they cannot measure time spent before the process started. Kubernetes documents that IfNotPresent skips a pull when an image is local, while Always still resolves the image and downloads only missing layers. That difference belongs in the first span, not in model loading.
Give every stage its own budget
Set an explicit upper bound for each stage and for their sum. Image pull covers registry resolution and missing layers. Artifact fetch covers tokenizer, configuration, shards, and auxiliary files for one immutable revision. Verification covers hashes, manifests, access checks, and any format validation. Loading covers host reads, deserialization, transformation, and placement into accelerator memory. Readiness covers warm-up work and the final probe transition.
A practical event record should contain start and end timestamps for every span, plus a terminal reason. Do not let a failed checksum appear as a slow fetch or let a readiness timeout appear as a load failure. NVIDIA Triton distinguishes a running server from a ready server through health endpoints, so routing should follow readiness rather than process liveness. The same separation applies to any serving stack, even when its probe names differ.
| Stage | Useful evidence | Typical owner |
|---|---|---|
| Image pull | Digest, cached layers, registry time | Platform |
| Artifact fetch | Revision, expected bytes, remote bytes | Model delivery |
| Verification | Manifest result, hash time | Supply chain |
| Load | Disk read time, host and device placement | Runtime |
| Readiness | Probe transition, warm-up result | Serving |
Compare warm NVMe with remote retrieval honestly
A local NVMe warm pool removes most remote artifact bytes from the critical path, but it does not remove verification, host reads, device placement, or warm-up. It also consumes node storage and creates an eviction problem. Remote retrieval avoids pre-positioning every model, but startup then depends on repository availability, network path, authentication, and concurrent downloads. Hugging Face documents a revision-aware cache that avoids downloading unchanged files, which illustrates why cache identity must include the model revision.
Measure three states separately: full hit for the exact revision, partial hit with some missing files, and miss. A directory existing is not proof of a hit. Confirm the manifest and complete file set before declaring residency. Pin immutable revisions instead of a mutable branch name, and keep image identity separate from model identity. A small image update should not force model weights to move, while a model revision change should not rely on an old directory that merely has the same friendly name.
Work a cold-start budget from bytes and spans
Consider an illustrative service with a 120 second readiness objective. Assume the image stage takes 8 seconds, verification 12 seconds, weight loading 38 seconds, and warm-up plus probe transition 10 seconds. That leaves 120 minus 8 minus 12 minus 38 minus 10, or 52 seconds for artifact retrieval. If the artifact set is 80 GB, the required sustained application-level rate is 80 GB divided by 52 seconds, or about 1.54 GB/s. These are hypothetical planning assumptions, not measured results.
Suppose observed remote retrieval at the tenth percentile is 0.8 GB/s under simultaneous scale-out. The fetch alone would take 80 divided by 0.8, or 100 seconds, producing an estimated 168 second cold start. A verified local copy that reads in 24 seconds would estimate 92 seconds overall. Actual transfer and storage performance varies by layout, contention, protocol, and hardware. The calculation decides whether remote-only startup is even compatible with the objective before anyone tunes the loader.
Use a measurable residency rule
Promote a model revision into the NVMe warm set when its miss penalty threatens the service objective and its reuse justifies the space. One rule is: retain the revision when its seven-day cold starts exceed two per node and its p90 remote-fetch span consumes more than 40 percent of the total readiness budget. Evict the lowest value revision only when free space would otherwise fall below a fixed reserve, such as 20 percent, and never evict a file set being loaded.
Evaluate the rule with a shadow calculation before automating deletion. Track avoided remote bytes, successful warm hits, partial hits, eviction followed by reload, and storage queue latency. If extra residency increases local read contention enough to lengthen loading, the warm pool is too dense or the array is shared badly. ClusterBid can help match storage and GPU capacity to the measured working set, but the input should be these traces rather than a generic model-size label.
Diagnose the slowest span in order
Start with the external timeline. If image pull expanded, inspect digest resolution, missing layers, registry errors, and node cache state. If artifact fetch expanded, compare expected with remote bytes, verify the requested revision, and check concurrency and authentication retries. If verification expanded, look for repeated hashing or scanning. If loading expanded, separate disk reads from CPU transformation and device copies. If readiness expanded after load, inspect warm-up, dependency checks, and probe configuration.
The method has limits. Span timing identifies where time accumulates, not automatically why. Operating-system page cache can make an apparent disk test a memory test, and aggregate bandwidth can conceal one slow shard. A warm pool also cannot guarantee readiness after a node replacement or cache eviction. Keep a tested remote path, exercise true misses deliberately, and size the cold-start objective for degraded but credible conditions rather than the best warm restart.