Move a release, not a filename
A file called model-final is a label, not an identity. Provider migration becomes reproducible when every deployable release has an immutable manifest. The manifest names each object, byte size, media type where useful, and cryptographic digest. It also records the model revision, serialization format, configuration, tokenizer, generation defaults, preprocessing assets, license or usage metadata required by policy, and the serving container digest.
Keep the human-friendly release name, but resolve it once to immutable identifiers. The Open Container Initiative uses descriptors containing media type, size, and digest, and recommends verifying retrieved content against the digest before consumption. The same pattern works for model bundles even when the destination object store has different paths. Location answers where the bytes are; the manifest answers which bytes are authorized for this release.
Bundle every behavioral dependency
Weights alone do not define inference behavior. A tokenizer vocabulary, merges or SentencePiece model, added tokens, special-token mapping, chat template, normalization rules, model configuration, generation configuration, and custom preprocessing code can change token IDs or outputs while the weight file remains identical. Save and hash them as one release unit. Hugging Face documents save_pretrained methods for model and tokenizer state, but the migration process still must inventory the actual files produced by the chosen classes.
Pin the runtime side too: container image by digest, framework and serving engine versions, CUDA libraries inside the image, and any custom kernels. Avoid treating a mutable container tag as evidence. Record the source repository commit or immutable revision, yet hash exported bytes separately because conversion, sharding, quantization, or serialization can produce a different artifact from the source checkpoint. The deployable bundle is the object being promoted.
Copy first, verify before registration
Stage the bundle under a new destination release ID. Transfer objects without exposing that ID to production. At the destination, verify object count, exact byte size, and a locally computed digest for every entry. Do not rely only on an upload success response or provider-specific checksum unless its algorithm and multipart semantics match the manifest. Parse configuration files and load the model in a quarantined environment only after byte verification passes.
Use a simple decision rule: promotion is blocked if any required path is missing, any size differs, any digest differs, or any unlisted executable file appears. This is intentionally strict. If a provider transforms bytes, such as recompressing an archive, define and hash the canonical unpacked objects rather than waiving verification. Keep the source bundle immutable until the destination release has completed production observation and rollback has expired.
Worked example: one sharded release
Assume an illustrative release contains four weight shards of 5 GB each, an index, model configuration, tokenizer JSON, special-token map, and generation configuration. The manifest contains nine objects: four shards and five metadata files. Treat GB as decimal units in this example, so the expected shard payload is 4 times 5 GB, or 20 GB, plus the small metadata files. The destination reports nine objects and approximately 20 GB, but that agreement is insufficient: two same-sized shards could be swapped or corrupted.
The verifier computes SHA-256 for every destination object and compares each digest with its named manifest entry. It then tokenizes a fixed corpus at source and destination and requires identical token ID arrays. Finally it loads a fixed request set and compares outputs under a defined deterministic configuration, or numeric tolerances where exact equality is not supported. These are illustrative checks, not proof of universal equivalence; nondeterministic kernels and hardware differences can require bounded statistical validation.
Validate in widening stages
After integrity comes compatibility. Run schema parsing, model load, tokenizer load, one CPU-side tokenization test, and a GPU smoke request on the destination stack. Then run a golden corpus that covers special tokens, multilingual or domain inputs, maximum accepted context, structured output, stop conditions, and known rejection cases. Compare tokenization separately from generation so a mismatch is localized rather than hidden in the final text.
Next shadow a bounded sample or replay sanitized traces without returning destination responses. Measure load failures, request success, time to first token, output rate, memory high-water mark, and application-level quality checks. Set gates before looking at results. An illustrative gate might require 100 percent tokenizer identity on the golden corpus, zero load errors, and latency within 10 percent of the approved baseline. Actual tolerances depend on the model contract and hardware.
Cut over with a reversible pointer
Promotion should change one controlled reference from the old immutable release to the new one. Keep routing weights, release IDs, and validation receipts in the change record. Start with a small traffic share, watch both technical and product error signals, then increase in explicit steps. Do not overwrite the previous bundle or retag it to mean something else. Rollback is a pointer change only if the old runtime, artifacts, credentials, and capacity remain usable.
Define rollback triggers and rehearse them. Trigger on digest or load failure immediately; use agreed thresholds for error rate, output validation, latency, or resource use. After rollback, verify that new requests resolve to the old release and that caches keyed only by a friendly model name are purged or versioned. Retain an audit record containing both manifests, transfer logs, validation results, routing steps, and the operator who approved promotion. Test restoration from the retained source after destination cleanup policies run, because a rollback pointer is useless if its objects have expired. If bandwidth is constrained, copy shards in parallel only within a tested limit and resume by verified object, never by assuming a partial file is valid. The limitation is important: hashes establish identity, not correctness. They prevent silent byte drift but cannot prove that a release is safe, accurate, or operationally compatible.