6.9k citations · 24.1k across the 46 of their papers we have counts for
5 papers · 1 filter
Scaling Laws for Autoregressive Generative Modeling
Tom Henighan, Jared Kaplan, Mor Katz +16
We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal imagetext models, and mathemat…
Learning to summarize from human feedback
Nisan Stiennon, Long Ouyang, Jeff Wu +6
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models…
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typicall…
Jukebox: A Generative Model for Music
Prafulla Dhariwal, Heewoo Jun, Christine Payne +3
We introduce Jukebox, a model that generates music with singing in the raw audio domain. We tackle the long context of raw audio using a multi-scale VQ-VAE to compress it to discre…
Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan +7
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute us…