62 citations · 73 across the 6 of their papers we have counts for
Showing 2023Show all
2 papers · 1 filter
cs.CL2023
Data Similarity is Not Enough to Explain Language Model Performance
Gregory Yauney, Emily Reif, David Mimno
Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this va…
cs.CL2023★ 7 cited
A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity
Shayne Longpre, Gregory Yauney, Emily Reif +8
Pretraining is the preliminary and fundamental step in developing capable language models (LM). Despite this, pretraining data design is critically under-documented and often guide…