Scaling Parameter-Constrained Language Models with Quality Data

Ernie Chang, Matteo Paltenghi, Yang Li +7

Scaling laws in language modeling traditionally quantify training loss as a function of dataset size and model parameters, providing compute-optimal estimates but often neglecting…