BiMix: A Bivariate Data Mixing Law for Language Model Pretraining

Ce Ge, Zhijian Ma, Daoyuan Chen +2

Large language models have demonstrated remarkable capabilities across various tasks, primarily attributed to the utilization of diversely sourced data. However, the impact of pret…