1k citations · 1.2k across the 7 of their papers we have counts for
8 papers
LIMA: Less Is More for Alignment
Chunting Zhou, Pengfei Liu, Puxin Xu +12
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and re…
A Theory on Adam Instability in Large-Scale Machine Learning
Igor Molybog, Peter Albert, Moya Chen +14
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant…
Effective Theory of Transformers at Initialization
Emily Dinan, Sho Yaida, Susan Zhang
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks a…
Scaling Laws for Generative Mixed-Modal Language Models
Armen Aghajanyan, Lili Yu, Alexis Conneau +7
Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens fr…
OPT: Open Pre-trained Transformer Language Models
Susan Zhang, Stephen Roller, Naman Goyal +16
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computatio…
Long-Term Planning and Situational Awareness in OpenAI Five
Jonathan Raiman, Susan Zhang, Filip Wolski
Understanding how knowledge about the world is represented within model-free deep reinforcement learning methods is a major challenge given the black box nature of its learning pro…
Dota 2 with Large Scale Deep Reinforcement Learning
OpenAI, :, Christopher Berner +24
On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as lo…
Neural Network Surgery with Sets
Jonathan Raiman, Susan Zhang, Christy Dennison
The cost to train machine learning models has been increasing exponentially, making exploration and research into the correct features and architecture a costly or intractable ende…