Scaling Data-Constrained Language Models

Niklas Muennighoff, Alexander M. Rush, Boaz Barak +6

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may so…