Dynamic Gradient Alignment for Online Data Mixing

Simin Fan, David Grangier, Pierre Ablin

The composition of training data mixtures is critical for effectively training large language models (LLMs), as it directly impacts their performance on downstream tasks. Our goal…