Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting

Xiang Yuan, Kaiqing Lei, Zhenyu Jin +3

The performance of Large Language Models (LLMs) is fundamentally influenced by the distributional composition of multi-domain pre-training data. While manual heuristics were preval…