1 citations · 1 across the 3 of their papers we have counts for
4 papers · 1 filter
Interleaved Head Attention
Sai Surya Duvvuri, Chanakya Ekbote, Rachit Bansal +6
Multi-Head Attention (MHA) is the core computational primitive underlying modern Large Language Models (LLMs). However, MHA suffers from a fundamental linear scaling limitation: $H…
Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
Rachit Bansal, Aston Zhang, Rishabh Tiwari +8
Progress on training and architecture strategies has enabled LLMs with millions of tokens in context length. However, empirical evidence suggests that such long-context LLMs can co…
The Art of Scaling Reinforcement Learning Compute for LLMs
Devvrit Khatri, Lovish Madaan, Rishabh Tiwari +6
Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pr…
Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
Bingbin Liu, Rachit Bansal, Depen Morwani +3
Diagonal preconditioners are computationally feasible approximate to second-order optimizers, which have shown significant promise in accelerating training of deep learning models.…