201 citations · 693 across the 61 of their papers we have counts for
34 papers · 1 filter
Continuous Chain of Thought Enables Parallel Exploration and Reasoning
Halil Alperen Gozeten, M. Emrullah Ildiz, Xuechen Zhang +3
Modern language models generate chain-of-thought traces by autoregressively sampling tokens from a finite vocabulary. While this discrete sampling has achieved remarkable success,…
Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
Yingcong Li, Davoud Ataee Tarzanagh, Ankit Singh Rawat +2
Linear attention methods offer a compelling alternative to softmax attention due to their efficiency in recurrent decoding. Recent research has focused on enhancing standard linear…
A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh +12
A primary challenge in large language model (LLM) development is their onerous pre-training cost. Typically, such pre-training involves optimizing a self-supervised objective (such…
A Statistical Framework for Data-dependent Retrieval-Augmented Models
Soumya Basu, Ankit Singh Rawat, Manzil Zaheer
Modern ML systems increasingly augment input instances with additional relevant information to enhance final prediction. Despite growing interest in such retrieval-augmented models…
Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyond
Yingcong Li, Ankit Singh Rawat, Samet Oymak
Recent research has shown that Transformers with linear attention are capable of in-context learning (ICL) by implementing a linear estimator through gradient descent steps. Howeve…
Mechanics of Next Token Prediction with Self-Attention
Yingcong Li, Yixiao Huang, M. Emrullah Ildiz +2
Transformer-based language models are trained on large datasets to predict the next token given an input sequence. Despite this simple training objective, they have led to revoluti…