3k citations · 8.1k across the 25 of their papers we have counts for
5 papers · 1 filter
Extracting Training Data from Large Language Models
Nicholas Carlini, Florian Tramer, Eric Wallace +9
It has become common to publish large (billion parameter) language models that have been trained on private datasets. This paper demonstrates that in such settings, an adversary ca…
Scaling Laws for Autoregressive Generative Modeling
Tom Henighan, Jared Kaplan, Mor Katz +16
We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal imagetext models, and mathemat…
Measuring the Algorithmic Efficiency of Neural Networks
Danny Hernandez, Tom B. Brown
Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to…
Language Models are Few-Shot Learners
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typicall…
Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan +7
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute us…