3k citations · 8.1k across the 25 of their papers we have counts for
Showing 2020 · cs.LGShow all
3 papers · 2 filters
cs.LG2020★ 150 cited
Scaling Laws for Autoregressive Generative Modeling
Tom Henighan, Jared Kaplan, Mor Katz +16
We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal imagetext models, and mathemat…
cs.LG2020★ 86 cited
Measuring the Algorithmic Efficiency of Neural Networks
Danny Hernandez, Tom B. Brown
Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to…
cs.LG2020★ 1.5k cited
Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan +7
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute us…