What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performance

Hong Meng Yam, Nathan J Paek

We explore the impact of pre-training data composition on the performance of small language models in a sample-efficient setting. Using datasets limited to 10 million words, we eva…