Object Storage for AI: S3-Compatible Data Lakes for GPU Datasets and Model Artefacts

Object storage is the foundation of the AI data stack. Training datasets (often 100 TB to 10+ PB) reside in object storage. Model weights at various precision formats and training stages are stored in object storage as immutable artefacts. Checkpoints during training are written to object storage for fault tolerance. Inference logs and output data accumulate in object storage for analysis and retraining.

The relationship between object storage and GPU compute is asymmetric: storage is written once and read many times (training data is consumed repeatedly across epochs), storage throughput must match GPU consumption rate (a GPU cluster consuming 100 GB/s of training data requires object storage that can deliver that throughput), and storage costs accumulate even when GPUs are idle (data persists between training runs and across model generations).

This post covers object storage architecture, performance optimisation for GPU workloads, cost tiering strategies, and integration patterns for AI infrastructure at mid-2026.

Object storage performance for AI is measured by: read throughput (GB/s sustained, critical for training data loading), read latency (time to first byte, impacts data loader startup), write throughput (GB/s for checkpoint writes, often the bottleneck in training resilience), and request rate (IOPS, matters for small-file datasets like image collections).

Standard S3 object stores deliver 5-20 GB/s per storage cluster, sufficient for small-to-medium training fleets. For large training clusters (256+ GPUs consuming 50+ GB/s of training data), performance optimizations are required: prefix-level sharding distributes requests across multiple storage nodes, client-side caching stores frequently accessed objects in local NVMe or parallel filesystem cache, and GPUDirect Storage (GDS) enables direct GPU-to-storage data transfer without CPU memory buffering.

At mid-2026, GPUDirect Storage is supported on H100 and B200 GPUs with compatible storage systems (WekaFS, Pure Storage, Dell PowerScale). GDS improves training data loading throughput by 20-40% for large-file workloads by eliminating the CPU memory copy that was previously required between storage and GPU.

AI object storage costs can be optimised through tiering based on data access frequency. The tiering strategy: hot tier (S3 Standard or equivalent) for active training datasets and current model checkpoints, accessed daily. Cost: $0.021-0.023/GB/month. Warm tier (S3 Infrequent Access) for completed training runs and checkpoints that may be needed for future runs. Cost: $0.010-0.012/GB/month. Cold tier (S3 Glacier or Deep Archive) for completed model versions, raw training data with retention requirements, and audit logs. Cost: $0.001-0.004/GB/month.

The table below shows the cost impact of tiering for a typical AI storage deployment. The savings are substantial: tiering reduces storage costs by 50-70% compared to keeping everything in hot storage.

The AI data lake architecture organises object storage into logical zones: raw zone (immutable source data as ingested, never modified), staging zone (data undergoing preprocessing, deduplication, filtering, and formatting), training zone (preprocessed training data organised by dataset version and split), and archive zone (completed training datasets with retention policy).

Data versioning is critical. Each training run should reference a specific dataset version, stored as an immutable object in the training zone. Version tags encode: data collection date, preprocessing pipeline version, dataset size in examples, and evaluation split hash. This enables reproducible training runs and controlled data experiments.

The data lake performance for training is dominated by the data loading pattern. PyTorch's DataLoader reads batches of training examples from object storage, typically using a mix of: local NVMe caching (copying training data to local SSDs before training for maximum throughput), and streaming from object storage (for datasets too large to cache locally, using multi-threaded prefetching). The standard practice at mid-2026 is to cache 10-20% of the training dataset locally and stream the remainder from object storage.

Model artefacts are stored in object storage with a structured naming convention. The artefact hierarchy: model family (llama, qwen, deepseek), model size (7B, 70B, 405B), training type (pretrain, sft, rlhf), precision format (fp16, fp8, int4), training timestamp (YYYY-MM-DD-HHMMSS), and artefact type (weights, tokenizer, config, merged).

Immutable storage is the standard practice: once a model artefact is written to object storage, it is never modified. A new training run produces new artefacts with new timestamps. This ensures reproducibility -- a deployment referencing a specific artefact path always loads the exact same weights. The immutability policy is enforced through object lock (write-once-read-many, WORM) on supported storage systems.

Artefact lifecycle automation regularly scans the artefact storage and transitions old artefacts to colder tiers. The policy: current training runs (versions < 30 days old) = hot tier. Recent completions (30-180 days) = warm tier. Historical versions (180+ days) = cold/archive tier. The transition is automated through storage lifecycle policies, reducing artefact storage costs by 40-60% without manual intervention.

Multi-provider GPU infrastructure requires object storage that is accessible from any GPU provider. The strategy is: use an S3-compatible object store (or cloud-agnostic provider like Cloudflare R2, Wasabi, or Backblaze B2) that is accessible from any GPU provider via standard S3 API, replicate training datasets to all GPU provider regions where training occurs (using storage replication or incremental sync), and ensure the storage layer is separate from the GPU compute provider (you can change GPU providers without migrating storage).

The storage-everywhere architecture: primary storage in a provider-neutral location (or the most cost-effective region), edge caching at each GPU provider location for frequently accessed datasets, data transfer orchestration (using rsync, Rclone, or storage replication) to keep edge caches synchronised, and checkpoint writes directed to both edge cache and primary storage for redundancy.

At mid-2026, the cost of cross-provider data transfer is $0.002-0.005/GB for intra-region transfers and $0.02-0.08/GB for inter-region transfers. A training run consuming 100 TB of data per epoch would incur $200-500/epocha in data transfer costs for intra-region, which is acceptable (typically 1-3% of total training cost). Inter-region costs are prohibitive for repeated data transfer, making edge caching essential.

Object storage sizing for AI infrastructure requires: training data volume (raw + preprocessed versions, including augmentation and synthetic data), model artefact volume (weights at multiple precision formats, checkpoints during training, model registry storage), growth rate (training data typically grows 20-40% per year, model artefact storage grows 50-100% per year as more models and versions are created), and performance requirements (read throughput matching GPU consumption rate, write throughput sufficient for simultaneous checkpointing from multiple training runs).

The capacity planning rule of thumb: allocate 3-5x the size of the largest training dataset for raw + staging + training zones, allocate 2-3x the size of the production model for artefact storage (accounting for multiple precision formats and training iterations), allocate checkpoint storage = number of concurrent training runs x checkpoint size x checkpoint count (30 checkpoints is typical for a multi-day training run).

For a 1,024-GPU cluster training large models: total storage volume typically reaches 5-15 PB within 12 months. At $0.01-0.02/GB/month blended tiered pricing, monthly storage cost is $50,000-300,000. This is 3-15% of the GPU compute cost, making storage a significant but manageable component of the AI infrastructure budget.