A Bitter Lesson for Data Filtering

Christopher Mohri, John Duchi, Tatsunori Hashimoto

We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that fil…