2.7k citations · 2.7k across the 4 of their papers we have counts for
4 papers
Effective Long-Context Scaling of Foundation Models
Wenhan Xiong, Jingyu Liu, Igor Molybog +18
We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with l…
Llama 2: Open Foundation and Fine-Tuned Chat Models
Hugo Touvron, Louis Martin, Kevin Stone +65
In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our f…
A Theory on Adam Instability in Large-Scale Machine Learning
Igor Molybog, Peter Albert, Moya Chen +14
We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant…
Over-parametrization via Lifting for Low-rank Matrix Sensing: Conversion of Spurious Solutions to Strict Saddle Points
Ziye Ma, Igor Molybog, Javad Lavaei +1
This paper studies the role of over-parametrization in solving non-convex optimization problems. The focus is on the important class of low-rank matrix sensing, where we propose an…