Image size is not a startup timer
A container image's advertised size is useful, but it cannot predict when a new GPU worker will accept traffic. The startup path includes manifest lookup, authentication, registry scheduling, transfer of missing compressed layers, digest verification, decompression, snapshot preparation, container creation, application initialization, model loading, GPU context creation, and a readiness decision. Several stages overlap, while others serialize on network, CPU, disk, or runtime locks. Treating one gigabyte as a fixed number of startup seconds hides the bottleneck that operators can actually change.
OCI defines an image as a manifest, configuration, and ordered filesystem layers. A descriptor carries a content digest and the byte size of the referenced content. A layer digest identifies its distributed bytes, while a DiffID identifies the uncompressed tar archive. This distinction matters operationally: registry bytes determine transfer work, but uncompressed contents and file layout influence verification, unpacking, and local storage work. Record both compressed bytes pulled and bytes written during unpack rather than relying on a registry UI's single total.
| Stage | Measure | Likely constraint |
|---|---|---|
| Resolve | manifest and auth latency | registry or identity service |
| Pull | missing compressed bytes and seconds | network or registry |
| Prepare | verify and unpack seconds | CPU, NVMe, runtime |
| Initialize | process to readiness seconds | application, model store, GPU |
Put four clocks around scale-out
Instrument timestamps at the scheduling decision, node image pull start and finish, container start, and readiness success. From these derive scheduler delay, artifact preparation, application initialization, and end-to-end capacity delay. Use runtime or kubelet events for image phases and an application-owned readiness event for the final boundary. A pod in Running state is not necessarily a serving worker, and a completed image pull says nothing about model weights obtained outside the image.
Keep distributions by node pool, image digest, registry region, and cache state. Do not mix a cached restart with a fresh node in one percentile. A practical report has at least p50 and p95 for cold pull, warm pull, unpack, model load, and readiness. Also report sample count and failure rate. With only a handful of scale-outs, a p95 looks precise while mostly describing one observation; retain raw event timelines for incident review.
Bound transfer before optimizing layers
An illustrative lower bound starts with missing compressed bytes divided by delivered throughput. Suppose a worker lacks 18 GiB, and each node receives 240 MiB/s after protocol overhead. Transfer alone is 18 x 1024 / 240 = 76.8 seconds. If registry policy or a shared link limits a 20-node wave to 1.2 GiB/s in aggregate, the same wave moves 360 GiB and has a fleet lower bound of 360 / 1.2 = 300 seconds. These are illustrative planning assumptions, not measured performance; actual throughput varies.
Run the same controlled pull on one clean node and on the intended scale-out wave. Capture registry response codes, per-layer timing, node network throughput, CPU use, and local disk writes. If one node meets the bound but the wave does not, investigate aggregate registry, NAT, or network limits. If transfer ends quickly but preparation continues, network tuning will not solve the delay. A measurable rule is to optimize transfer only when pull time exceeds half of the cold artifact-preparation interval at p95.
Layer reuse is a fleet property
Content-addressed layers let a runtime avoid fetching blobs already present locally, but reuse depends on node history and exact digests. A stable base layer shared across services can help established nodes while doing nothing for newly provisioned nodes. Rebuilding an early layer may also invalidate later build cache during image construction, although runtime reuse still follows the emitted content digests. Inspect the manifest and node content store instead of assuming that similar Dockerfiles produce reusable fleet artifacts.
Separate rarely changing CUDA and operating-system content from frequently changing application code when that split matches ownership and security patching. Remove build tools and intermediate outputs with multi-stage builds. Do not create dozens of tiny layers merely to chase reuse: every design still needs operational testing, and file count or unpack behavior can dominate even when compressed bytes fall. Publish image digests, not mutable tags, in measurements so a later build cannot silently change the subject.
Control concurrency, not just bandwidth
Kubernetes documents that kubelet can pull images serially or in parallel and that maxParallelImagePulls can protect network bandwidth and disk I/O when serializeImagePulls is false. The maximum-parallel-pulls field became stable in Kubernetes 1.35, so older deployed releases require their own documented configuration and feature-gate check. Registry-side quotas, authentication throttles, and retry backoff can create a second concurrency ceiling. Unlimited simultaneous pulls may reduce the first pod's delay yet extend the last pod's delay and increase failures.
Test waves of 1, 5, 10, and 20 fresh nodes, then plot time until the first, half, and all workers are ready. A useful decision rule is to stop increasing pull concurrency when p95 readiness improves by less than 10 percent or pull failures exceed 1 percent in the test wave. Those thresholds are explicit local policy, not vendor defaults. Preserve headroom for normal deployments and rollbacks rather than benchmarking at the registry's failure edge.
Cache artifacts without faking readiness
Node pre-pulls, golden images, and local registry mirrors can reduce missing artifact work. They should be evaluated with cache hit ratio, cache age, eviction rate, and the time to repopulate a replaced node. Pinning every large image can exhaust local storage and turn garbage collection into the next outage. A pre-pull also needs an integrity check against the intended digest; the presence of an old tag is not proof that the correct bytes are available.
Keep artifact availability separate from application readiness. A worker should become ready only after its model revision, tokenizer or auxiliary files, GPU context, and a minimal functional check are complete. When comparing capacity through ClusterBid, match providers to the measured cold-node network, local NVMe, and readiness requirements rather than comparing GPU count alone. The honest limitation is that caching improves repeated starts, but it cannot guarantee a first start on a replaced or newly autoscaled node.
Adopt separate cold and warm budgets
Define two service objectives for the provisioning path. A warm budget covers an existing node with all required image layers present; a cold budget covers a node with no relevant layers. For example, an illustrative policy might require 95 percent of warm workers to become ready within 70 seconds and 95 percent of cold workers within 240 seconds, measured from scheduling decision to readiness. The numbers must come from demand tolerance and observed distributions, not from this example.
Review the budgets whenever image digest, runtime, node image, registry path, or model-loading procedure changes. Fail a release candidate if its controlled cold p95 rises by more than 15 percent across at least 20 comparable trials, while also inspecting confidence and raw values. Track registry errors and readiness failures beside latency so a fast failed pull cannot improve the chart. This closes the loop: smaller images are valuable when they reduce a measured stage without moving delay or failure into another one.