296 citations · 1.7k across the 43 of their papers we have counts for
5 papers · 1 filter
Mimetic Initialization Helps State Space Models Learn to Recall
Asher Trockman, Hrayr Harutyunyan, J. Zico Kolter +2
Recent work has shown that state space models such as Mamba are significantly worse than Transformers on recall-based tasks due to the fact that their state size is constant with r…
Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count
Hanseul Cho, Jaeyoung Cha, Srinadh Bhojanapalli +1
Transformers often struggle with length generalization, meaning they fail to generalize to sequences longer than those encountered during training. While arithmetic tasks are commo…
Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure
Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi +3
Even for simple arithmetic tasks like integer addition, it is challenging for Transformers to generalize to longer sequences than those encountered during training. To tackle this…
Efficient Language Model Architectures for Differentially Private Federated Learning
Jae Hun Ro, Srinadh Bhojanapalli, Zheng Xu +2
Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the s…
HiRE: High Recall Approximate Top- Estimation for Efficient LLM Inference
Yashas Samaga B L, Varun Yerram, Chong You +4
Autoregressive decoding with generative Large Language Models (LLMs) on accelerators (GPUs/TPUs) is often memory-bound where most of the time is spent on transferring model paramet…