DAOS Architecture
DAOS is a parallel storage system providing POSIX, NVMe-native, Intel Optane support. The storage cluster delivers 300+ GB/s read/write of throughput through parallel architecture. Key design features include: POSIX, NVMe-native, Intel Optane support protocol support, erasure coding or replication for data durability, and integration with major GPU cluster orchestration platforms.
Performance Benchmarks for AI
Performance benchmarks for DAOS: sequential read throughput of 300+ GB/s read/write with 1M+ IOPS; metadata operations at 50K-500K ops/second depending on namespace design; checkpoint write throughput of 30-70% of peak read due to parity/replication overhead. For checkpointing a 70B model (140 GB) with 32 GPUs: checkpoint save time of 2-10 seconds with GPUDirect Storage vs 20-60 seconds without.
Cost Analysis and Budgeting
DAOS costs $70-$180/TB-month on DAOS on-premise, cloud via OCI. For a 64-GPU cluster with 100 TB of active storage for training data and checkpoints: monthly storage cost of $2,000-$25,000 depending on performance tier. Total storage as percentage of cluster TCO: 10-30% for parallel filesystems, 5-15% for object storage, 2-5% for local NVMe.
GPU Cluster Integration
Integration with GPU clusters: Kubernetes CSI driver for container-native storage access; Slurm job integration with filesystem mounts; GPUDirect Storage support for direct GPU<->filesystem DMA; and data lifecycle policies for hot/warm/cold tiering. Training frameworks (PyTorch, NeMo, Megatron-LM) access DAOS via standard filesystem interfaces or optimized data loaders.
Deployment Patterns
Deployment of DAOS for AI clusters: dedicated storage cluster with GPU-direct networking; all-flash NVMe storage nodes for maximum throughput; storage nodes connected via InfiniBand or high-speed Ethernet to compute cluster; and data staging from object store to parallel filesystem for training jobs. For multi-cluster deployments, global namespace and cross-region data replication are critical.
Recommendations and Best Practices
Best practices for DAOS: provision 30-50% more throughput than peak checkpoint load to avoid training stalls; separate checkpoint and dataset storage for independent scaling; implement data caching on local NVMe to reduce parallel filesystem load; use GPUDirect Storage for 2-5x checkpoint speedup; monitor filesystem metadata performance separately from data throughput; and plan for 50% annual capacity growth.