3k citations · 8.1k across the 25 of their papers we have counts for
7 papers · 1 filter
In-context Learning and Induction Heads
Catherine Olsson, Nelson Elhage, Neel Nanda +23
"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect ev…
Scaling Laws and Interpretability of Learning from Repeated Data
Danny Hernandez, Tom Brown, Tom Conerly +15
Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or uninte…
Scaling Laws for Autoregressive Generative Modeling
Tom Henighan, Jared Kaplan, Mor Katz +16
We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal imagetext models, and mathemat…
Measuring the Algorithmic Efficiency of Neural Networks
Danny Hernandez, Tom B. Brown
Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to…
Scaling Laws for Neural Language Models
Jared Kaplan, Sam McCandlish, Tom Henighan +7
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute us…
Transfer of Adversarial Robustness Between Perturbation Types
Daniel Kang, Yi Sun, Tom Brown +2
We study the transfer of adversarial robustness of deep neural networks between different perturbation types. While most work on adversarial examples has focused on and…
Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
Nicolas Papernot, Fartash Faghri, Nicholas Carlini +23
CleverHans is a software library that provides standardized reference implementations of adversarial example construction techniques and adversarial training. The library may be us…