201 citations · 693 across the 61 of their papers we have counts for
7 papers · 1 filter
Large Language Models with Controllable Working Memory
Daliang Li, Ankit Singh Rawat, Manzil Zaheer +5
Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP), owing to their excellent understanding and generation abilities. Remarkably…
Generalization Properties of Retrieval-based Models
Soumya Basu, Ankit Singh Rawat, Manzil Zaheer
Many modern high-performing machine learning models such as GPT-3 primarily rely on scaling up models, e.g., transformer networks. Simultaneously, a parallel line of work aims to i…
A Fourier Approach to Mixture Learning
Mingda Qiao, Guru Guruganesh, Ankit Singh Rawat +2
We revisit the problem of learning mixtures of spherical Gaussians. Given samples from mixture , the goal is to estimate the means $…
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
Zonglin Li, Chong You, Srinadh Bhojanapalli +8
This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the interm…
Teacher Guided Training: An Efficient Framework for Knowledge Transfer
Manzil Zaheer, Ankit Singh Rawat, Seungyeon Kim +5
The remarkable performance gains realized by large pretrained models, e.g., GPT-3, hinge on the massive amounts of data they are exposed to during training. Analogously, distilling…
ELM: Embedding and Logit Margins for Long-Tail Learning
Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat +1
Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners. Several recent approaches for the problem have propose…