FastMix: Fast Data Mixture Optimization via Gradient Descent

Haoru Tan, Sitong Wu, Yanfeng Chen +5

While large and diverse datasets have driven recent advances in large models, identifying the optimal data mixture for pre-training and post-training remains a significant open pro…