Language Models Improve When Pretraining Data Matches Target Tasks

David Mizrahi, Anders Boesen Lindbo Larsen, Jesse Allardice +7

Every data selection method inherently has a target. In practice, these targets often emerge implicitly through benchmark-driven iteration: researchers develop selection strategies…