Storage Bills Can Outlast the GPU Training Run

A training cluster may shut down on Friday while its checkpoints, optimizer states, logs, evaluation outputs, container layers, and copied datasets remain for months. The correct budget horizon is therefore the artifact lifetime, not the GPU reservation. Start with bytes by artifact class, copy count, growth per run, retention state, and expected retrieval or transfer activity.

Experiment systems can split one logical run across several stores. MLflow, for example, documents a backend store for parameters, metrics, and tags and a separate artifact store for larger files such as model weights and data. Deleting a run from normal views does not necessarily remove its metadata or artifacts automatically. An inventory that counts only the visible model directory can miss both supporting history and recoverable deleted runs.

Trace an artifact from creation to deletion. A checkpoint may first land on local NVMe, copy to shared storage, replicate into object storage, be downloaded for evaluation, and be retained again with a registered model. Logs may exist in the training framework, cluster logging system, experiment tracker, and an analyst export. Label each location as authoritative, cache, replica, derivative, or unknown.

Give every class a named owner and a deletion authority. The training team may know which intermediate checkpoints are obsolete, while the platform team controls lifecycle rules and the model owner defines recovery needs. Without that split, nobody can safely delete anything. Unknown ownership should trigger quarantine and review, not an automatic purge.

Use monthly byte-hours or the billing unit specified by the storage service. For each class, calculate average retained quantity multiplied by unit price, then add request, retrieval, early-deletion, replication, and egress assumptions separately. Do not assume that a delete immediately eliminates all chargeable versions. AWS documents that expiration behaves differently for unversioned and versioned buckets and that noncurrent versions require their own expiration action.

Set budget checkpoints at useful events: experiment creation, first successful recovery checkpoint, job completion, model promotion, and retention expiry. At each event, compare actual bytes and object counts with the forecast. A percentage alert alone is weak when the baseline is tiny, so combine it with an absolute threshold. For example, review when either forecast is exceeded by 20 percent or unowned data exceeds 500 GB. Those thresholds are illustrative local policy.

Consider an illustrative run that writes twelve 400 GB checkpoints, 300 GB of logs and evaluations, and one 400 GB final model. It also keeps a second durable copy of the final model. The initial retained quantity is 12 multiplied by 400 GB, plus 300 GB, plus 400 GB, plus another 400 GB, for 5,900 GB or 5.9 TB using decimal units. Actual storage prices and compression ratios vary.

Assume a hypothetical $0.020 per GB-month for straightforward arithmetic, not a live quote. Holding 5,900 GB for three months costs 5,900 multiplied by $0.020 multiplied by 3, or $354, before requests, retrieval, transfer, taxes, or minimum-duration effects. If recovery tests show that only the newest three of the twelve intermediate checkpoints are needed after week one, deleting the other nine reduces retained quantity by 9 multiplied by 400 GB, or 3,600 GB. For this example only, treat one week as 0.25 month. Over the remaining 2.75 months, the illustrative avoided storage charge is 3,600 multiplied by $0.020 multiplied by 2.75, or $198.

Retention should answer a recovery question. Keep the checkpoint required to resume the active run, the final artifact required to reproduce deployment, and enough logs and metadata to explain the result. Before deleting older checkpoints, restore one of the retained candidates into an isolated environment and verify that model, optimizer, scheduler, random-state, code revision, and data references meet the team's recovery objective.

Use a decision rule: an artifact may leave the protected retention set only when an owner identifies its replacement, the replacement passes the documented recovery test, and the required investigation window has closed. If any condition is unknown, keep it temporarily and create a dated review item. This is slower than age-only cleanup but avoids paying forever for files nobody trusts or deleting the only usable restart point.

When a cleanup job reports success but billed storage stays high, check in order. First list current objects. Second list noncurrent versions and delete markers. Third inspect incomplete multipart uploads and replicas. Fourth compare experiment-tracker deletion state with the underlying artifact store. Fifth confirm lifecycle eligibility and event timing. Finally reconcile the provider's reporting interval with the cleanup timestamp.

AWS notes that lifecycle expiration of a current object in a versioned bucket can create a delete marker while preserving the former current version as noncurrent. MLflow notes that ordinary run deletion does not automatically remove backend metadata or artifact data. Those documented mechanisms are diagnostic possibilities, not proof of the cause in another system. Preserve inventories before attempting permanent deletion.

Run a weekly report grouped by project, artifact class, age, owner, protection state, and last successful restore. Require teams to resolve unknown owners and expired exceptions. A useful service objective is that at least 95 percent of retained bytes have both an owner and a future review date, with the exact target chosen by the organization. Report storage after compute termination as part of the run's total cost receipt.

The limitation is that the cheapest-looking retention policy may fail a rare investigation or reproducibility need. Storage classes can also impose retrieval latency or minimum-duration charges, and lifecycle execution is not always immediate. Keep pricing inputs external to the article and validate them against the chosen service. Cleanup ownership should follow durable recovery requirements, not a blanket desire to make the graph fall.

Expiring objects, Amazon Simple Storage Service User Guide

Artifact Stores, MLflow documentation