Exiting a GPU Provider Without Losing Your Experiment History

An exit is complete only when the team can answer the questions its old environment answered. That usually requires run identifiers, parameters, metrics, timestamps, tags, code and container references, dataset identifiers, checkpoints, model artifacts, evaluation outputs, logs, and lineage between them. A directory full of weights is not experiment history if nobody can connect a file to the run that produced it.

Separate the logical inventory from physical locations. MLflow documents that run metadata lives in a backend store while large artifacts live in an artifact store. Other systems may make a similar split or expose only an API. Record the system of record, export method, format, owner, byte and object count, retention obligation, and destination for each class before copying begins.

Search manifests and metadata for provider-specific object URIs, registry names, image repositories, identity principals, encryption-key references, secret locations, job IDs, and private endpoints. Also identify managed databases or dashboards whose useful state cannot be reproduced from object storage alone. A successful file copy will not repair an experiment record that still points to an unreachable URI.

Classify every dependency as portable, transformable, replaceable, or intentionally retired. Preserve original identifiers in a mapping table even when the destination assigns new ones. That table should connect source run ID to destination run ID, source artifact URI to destination URI, and old code or image digest to its retained equivalent. Never put secret values in the mapping.

Use documented APIs or provider-supported export paths rather than scraping a user interface. Export immutable content first, then mutable metadata close to cutover. Create a manifest containing relative path or object key, byte size, checksum where available, source version ID, content type, export timestamp, and the logical run ID. Preserve the tool version and command options used to create the export.

MLflow's artifact API, for example, documents list and download operations for an artifact URI or run ID. Those calls help enumerate and retrieve artifacts but do not by themselves export a tracking database or prove semantic completeness. If the platform has distinct metadata and artifact stores, capture both through their supported procedures and test restoration against a compatible version.

Suppose an illustrative inventory contains 2,400 runs, 18,600 artifact objects, and 7.200 decimal TB of content. The first export reports 2,400 run records and 18,592 objects totaling 7.199 decimal TB. That is not close enough merely because the percentages look high. The byte shortfall is 0.001 decimal TB, which equals 1 decimal GB. Reconcile the eight missing objects and the exact byte difference by key and size before declaring success.

Assume six missing objects are zero-byte completion markers intentionally excluded by the export tool and two are checkpoint shards, totaling 1 decimal GB, that failed during transfer. Record the six exclusions in the manifest and re-copy the two shards. The acceptance calculation is then 18,594 expected transferable objects minus 18,594 verified destination objects equals zero unexplained objects, with verified bytes also equal to the transferable manifest total. Counts are illustrative; actual estates vary.

Checksums and counts establish transport integrity, not usability. Select a stratified sample containing a recent successful run, an older run, a failed run, a registered or promoted model, a large sharded checkpoint, and an experiment with unusual tags. In the destination, search the metadata, render metric history, download artifacts, load the model or checkpoint in an isolated environment, and follow dataset and code references.

A measurable decision rule is: do not cut over until object and byte reconciliation has zero unexplained differences, every critical artifact passes checksum verification, and all predefined reconstruction tests pass. For noncritical history, choose and document a sampling target, such as 30 runs across age and workload classes. That number is illustrative and provides evidence, not certainty about every record.

Freeze structural changes, perform the bulk copy, then capture the delta created during transfer. Move a small group of new jobs to the destination while the old environment remains readable. Compare run creation, metric logging, artifact upload and download, model registration, checkpoint restart, and incident troubleshooting. Publish the point after which the destination becomes authoritative.

Keep a time-bounded rollback window with explicit conditions. A rollback should mean routing new work back through a tested path, not merging two uncontrolled writers. During the window, mark source data read-only where supported and prevent lifecycle rules from deleting required evidence. ClusterBid can help match replacement GPU capacity to measured workload requirements, but the experiment-history acceptance test remains the team's responsibility.

After the acceptance record is signed, revoke old human accounts, service principals, API tokens, SSH keys, registry credentials, storage grants, federation links, and automation webhooks through the organization's normal access-management process. Rotate credentials that were ever present in the old environment. Remove network routes and scheduled jobs only after confirming that the destination no longer depends on them.

Use a sequence that can be audited: stop new writes, verify the final delta, snapshot the inventory, obtain owner approval, revoke nonessential access, monitor for failed calls, then close remaining administrative access under the provider's standard procedure. Do not improvise destructive commands or bypass retention controls. Provider-side deletion and account closure should follow the contract and established organizational approvals.

Store the final manifest, checksum results, identifier mapping, reconstruction results, exclusions, approval, revocation evidence, and retention dates somewhere independent of the former provider. Record which artifacts were intentionally omitted and why. A later auditor or engineer should be able to distinguish a planned exclusion from silent loss without reopening the old account.

The limitation is that proprietary dashboards, scheduler state, managed keys, or service-specific metadata may not have a portable equivalent. Exporting screenshots does not make those features executable. State the residual loss plainly and retain raw data where permitted. The goal is not a perfect clone of the provider; it is verified continuity for the experiment questions and recovery actions the organization has chosen to preserve.

Artifact Stores, MLflow documentation

mlflow.artifacts Python API

Rubin R100 GPU Procurement Guide 2026: What AI Teams Should Buy Now vs Wait For

AI Audio and Speech Model Deployment: Whisper, ElevenLabs, Bark GPU Inference