ADMIRE-BayesOpt: Accelerated Data MIxture RE-weighting for Language Models with Bayesian Optimization

Shengzhuang Chen, Xu Ouyang, Michael Arthur Leopold Pearce +2

Determining the optimal data mixture for large language model training remains a challenging problem with an outsized impact on performance. In practice, language model developers…