Curriculum Learning at Scale: Training Data Sequencing and GPU Optimization

The intuition behind curriculum learning is straightforward: training on easy examples first, then progressively harder ones, mirrors how humans learn. In practice, the results are mixed. For LLMs pretrained on web-scale data, random ordering is surprisingly effective because the data distribution naturally contains a mix of difficulties at every step. However, for fine-tuning on domain-specific tasks - code generation, mathematical reasoning, long-context understanding - curriculum strategies consistently show 2-5 percent accuracy improvements and 1.5-3x convergence speedups.

The GPU infrastructure question is whether the compute cost of scoring, sorting, and dynamically scheduling data is worth the training efficiency gains. At 1 trillion tokens of LLM pretraining, a single pass of inference-based difficulty scoring on 10 percent of tokens costs $5,000-15,000 in GPU compute. The return must be measured in reduced training steps - even a 5 percent reduction in steps to target loss on a $2M training run saves $100,000.

Three families of difficulty scoring exist. Heuristic scoring (length, perplexity from a smaller reference model, vocabulary rarity) requires only CPU preprocessing at near-zero GPU cost. The perplexity approach runs each training example through a small reference model (GPT-2 124M) on GPU, costing approximately $0.50-1.00 per million tokens. For a 1T token corpus, this is $500-1,000 - negligible for most training pipelines.

Inference-based scoring uses the target model itself (or a precursor checkpoint) to compute loss on each example. This is up to 100x more expensive because it requires a forward pass through the full-size model. For a 70B model on 100M held-out tokens, scoring costs $2,000-5,000. The benefit: this captures the model-specific difficulty distribution, which correlates strongly with learning outcomes. Companies like Mistral and Meta reportedly use a hybrid: heuristic pre-filtering reduces the candidate pool by 90 percent, then inference-based scoring on the remaining 10 percent.

Once data is scored, the curriculum schedule determines how difficulty increases over training. Fixed pacing defines difficulty thresholds a priori (e.g., steps 0-10K: bottom 20 percent difficulty, steps 10K-100K: bottom 50 percent, steps 100K+: all data). This requires a single pre-processing pass and imposes zero runtime GPU overhead. The data loader simply filters by threshold per step, streaming from disk.

Dynamic pacing adjusts difficulty based on the model's current performance. The model's loss on a held-out easy set and hard set determines when to advance the difficulty threshold. This requires periodic evaluation passes - typically every 500-1000 steps - each costing 5-15 minutes of GPU inference on the evaluation sets. For a 70B model training for 100K steps, dynamic pacing adds 500-1500 GPU-hours of evaluation compute, approximately $1,500-4,500. The convergence gain (typically 5-15 percent fewer steps) easily offsets this cost.

Online curriculum goes further by scoring batches during training using the current model state. At each step, the data sampler presents multiple candidate batches from different difficulty buckets, and the model's loss on each batch determines which one to use. This adds 2-3x data loading overhead and requires the data loader to maintain multiple streams of pre-scored candidates in GPU memory.

A consistent finding across curriculum learning research is that curriculum matters most during the first 10-20 percent of training. After the model has seen enough data to form reasonable representations, data ordering provides diminishing returns. The practical recommendation: invest GPU budget in curriculum for the first 20-50K steps (or first 10-20 percent of tokens), then switch to random ordering or anti-curriculum (hard examples first) for the remainder.

Anti-curriculum - training on the hardest examples first - has shown surprising effectiveness for supervised fine-tuning (SFT) and RLHF. The intuition is that SFT data is curated and relatively clean; exposing the model to edge cases early builds robust decision boundaries. For an RLHF preference tuning run of 10K steps, anti-curriculum with inference-based scoring adds $200-500 in scoring cost but reportedly achieves the same reward model score in 6,000-7,000 steps, saving $3,000-8,000 in training GPU cost.

Dynamic curriculum data loading requires multiple data streams. The GPU memory overhead for curriculum-aware data loading is modest: approximately 200-500 MB for storing scored difficulty metadata for the current shard plus 2-4 candidate batches (each 8-16 tokens x batch size sequences). The main overhead is on the CPU data loading side, where the dataloader must maintain sorted indices and slice by difficulty threshold.

NVIDIA's DALI and PyTorch's DataLoader with custom Sampler subclasses handle the streaming logic. The key optimization is pre-bucketing: data is pre-sorted into 10-20 difficulty buckets during preprocessing, and each bucket is stored as a separate binary file. The GPU dataloader streams from the appropriate bucket based on the current curriculum step, requiring no on-the-fly sorting. Pre-bucketing adds approximately 10-20 percent to preprocessing time but zero runtime overhead.

B200's larger memory enables storing multiple difficulty-sorted shards in GPU-accessible memory (via GPUDirect Storage or unified memory), reducing the CPU-to-GPU data transfer bottleneck that limits curriculum throughput. With 192 GB VRAM, approximately 40-60 GB can be allocated to pre-loaded curriculum data, enabling instant switching between difficulty levels without disk reads.

For online curriculum methods that score multiple candidate batches per step, B200's faster HBM3e bandwidth (8 TB/s) reduces the batch evaluation overhead by 30-40 percent. The estimated throughput improvement for online curriculum training on B200 versus H100 is 1.3-1.5x, making previously impractical online methods feasible for production training pipelines.