FSDP2 GPU Requirements
FSDP2 PyTorch native by Meta/PyTorch supports GPU training and inference with features: Fully Sharded Data Parallellism, per-parameter sharding, computation/communication overlap, HF integration... GPU requirements: NVIDIA GPUs with compute capability 8.0+ (Ampere/Hopper/Blackwell), CUDA 12.x+, and NCCL 2.20+ for multi-GPU. ROCm support is available for AMD MI-series GPUs. Minimum 16 GB VRAM recommended for model development, 80 GB+ for production.
Installation and Configuration
Installation: via pip/conda with CUDA/ROCm variant selection. GPU configuration: GPU count, precision mode (FP32/FP16/BF16/FP8), data parallel vs model parallel strategy, and memory allocation limits. Configuration flags for distributed training: world size, rank, master address/port, backend (NCCL/GLOO), and timeout settings.
Multi-GPU Scaling
Multi-GPU scaling with FSDP2: supported parallelism strategies including data parallel for smaller models, tensor parallel for large models that exceed single-GPU memory, pipeline parallel for multi-node deployments, and hybrid parallel (3D parallelism) for frontier-scale training. Communication backend: NCCL for NVIDIA GPUs, RCCL for AMD. Bandwidth requirements: intra-node NVLink preferred, inter-node InfiniBand/RoCEv2.
Performance Tuning
Performance optimization for FSDP2: enable automatic mixed precision (AMP) for 40-70% training speedup; use torch.compile/JIT/XLA for kernel fusion; tune batch size for optimal memory-throughput tradeoff; enable gradient checkpointing for 30-50% memory reduction; configure data loading pipeline (num workers, prefetch factor, pinned memory); and benchmark with profiler tools to identify bottlenecks.
Production Deployment
Production patterns for FSDP2: containerized deployment with Docker + NVIDIA Container Toolkit; Kubernetes with node/pod affinity for GPU topology; model versioning and registry integration; CI/CD pipeline with GPU testing on pre-production instances; monitoring with Prometheus/DCGM; and autoscaling policies based on GPU utilization and queue depth.
Troubleshooting and Common Issues
Common issues with FSDP2: CUDA out of memory (adjust batch size, enable checkpointing, use gradient accumulation); NCCL timeout (check network latency, ring order, IB/RoCE configuration); NaN losses (reduce learning rate, check mixed precision settings, validate input data); slow data loading (increase num_workers, enable prefetching, use NVMe storage); and model hang/deadlock (verify distributed setup, check barrier synchronization).