1k citations · 1.2k across the 7 of their papers we have counts for
4 papers · 1 filter
LIMA: Less Is More for Alignment
Chunting Zhou, Pengfei Liu, Puxin Xu +12
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and re…
A Theory on Adam Instability in Large-Scale Machine Learning
Igor Molybog, Peter Albert, Moya Chen +14
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant…
Effective Theory of Transformers at Initialization
Emily Dinan, Sho Yaida, Susan Zhang
We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks a…
Scaling Laws for Generative Mixed-Modal Language Models
Armen Aghajanyan, Lili Yu, Alexis Conneau +7
Generative language models define distributions over sequences of tokens that can represent essentially any combination of data modalities (e.g., any permutation of image tokens fr…