Data Similarity is Not Enough to Explain Language Model Performance

Gregory Yauney, Emily Reif, David Mimno

Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this va…