IMU-1: Sample-Efficient Pre-training of Small Language Models

George Grigorev

We present IMU-1, a 430M-parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data. We describe a validated traini…