Full Fine-Tuning (SFT) Overview
Full Fine-Tuning (SFT) is Full parameter training. It costs approximately 100% (full cost) compared to full parameter fine-tuning. The technique requires 8-32 GPUs (H100/B200) GPUs with training time of 12-72 hours. Key tools: NVIDIA NeMo, Megatron-LM, HuggingFace. Memory savings come from reducing trainable parameter count while maintaining model quality for specific tasks.
GPU Requirements
Recommended GPU configuration for Full SFT: 8-32 GPUs (H100/B200). For a 7B model, VRAM usage is approximately: 14 GB for base model weights (FP16), 0.5-4 GB for adapter weights, 8-16 GB for optimizer states, 2-4 GB for activations (with gradient checkpointing). Total: 25-38 GB, fitting on single A100 80GB or L40S 48GB. For 70B models with QLoRA: 70 GB base (INT4) + 2-8 GB adapters + 4-8 GB optimizer = 76-86 GB, requiring 2x A100 80GB or 1x H100 80GB with offloading.
Training Time and Cost
Expected training time: 12-72 hours. Cloud GPU cost estimate per run on A100 80GB: $28-$84. Cost on H100: 30-50% higher per GPU-hour but typically 20-40% faster training time, making cost-per-run similar or slightly lower. For production pipelines with daily retraining, monthly costs range from $1015-$3051.
Quality vs Speed Tradeoffs
Quality metrics for Full SFT vs full fine-tuning: typically within 1-3% of full SFT quality on downstream tasks. Resource savings: 100% (full cost) of the compute cost. Speed advantage: 5-10x faster time-to-quality than full fine-tuning. Best suited for: domain adaptation, instruction tuning, personalization, and task-specific optimization.
Production Pipeline Design
Production Full SFT pipeline: data preparation and quality filtering; base model selection and quantization; hyperparameter optimization (rank, alpha, learning rate); training with early stopping and checkpointing; model evaluation on holdout set; adapter merging or LoRA stacking for deployment; and A/B testing against baseline. CI/CD integration with automated GPU provisioning and cost tracking.
Cost Optimization
Optimize Full SFT costs: use spot/preemptible GPUs with checkpointing (60-80% savings); select optimal batch size for GPU memory utilization; use gradient accumulation for effective larger batches; enable gradient checkpointing for memory savings; implement automatic mixed precision (BF16/FP16); and schedule training during off-peak pricing periods.