Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them

Kevin Zhou, Lisa Alazraki, Kris Cao +1

Pre-training data mixtures are commonly tuned by running small-scale experiments and extrapolating to the target training budget. When high-quality data is scarce and must be repea…