3k citations · 3.7k across the 6 of their papers we have counts for
Showing cs.LGShow all
3 papers · 1 filter
cs.LG2022★ 87 cited
In-context Learning and Induction Heads
Catherine Olsson, Nelson Elhage, Neel Nanda +23
"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect ev…
cs.LG2022★ 22 cited
Scaling Laws and Interpretability of Learning from Repeated Data
Danny Hernandez, Tom Brown, Tom Conerly +15
Recent large language models have been trained on vast datasets, but also often on repeated data, either intentionally for the purpose of upweighting higher quality data, or uninte…
cs.LG2020★ 150 cited
Scaling Laws for Autoregressive Generative Modeling
Tom Henighan, Jared Kaplan, Mor Katz +16
We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal imagetext models, and mathemat…