Olmix: A Framework for Data Mixing Throughout LM Development

Mayee F. Chen, Tyler Murray, David Heineman +5

Data mixing -- determining the ratios of data from different domains -- is a first-order concern for training language models (LMs). While existing mixing methods show promise, the…