Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan +7

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute us…