463 citations · 1.5k across the 29 of their papers we have counts for
8 papers · 1 filter
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Daniel Y. Fu, Tri Dao, Khaled K. Saab +3
State space models (SSMs) have demonstrated state-of-the-art sequence modeling performance in some modalities, but underperform attention in language modeling. Moreover, despite sc…
Transform Once: Efficient Operator Learning in Frequency Domain
Michael Poli, Stefano Massaroli, Federico Berto +4
Spectral analysis provides one of the most effective paradigms for information-preserving dimensionality reduction, as simple descriptions of naturally occurring signals are often…
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
Eric Nguyen, Karan Goel, Albert Gu +5
Visual data such as images and videos are typically modeled as discretizations of inherently continuous, multidimensional signals. Existing continuous-signal models attempt to expl…
ButterflyFlow: Building Invertible Layers with Butterfly Matrices
Chenlin Meng, Linqi Zhou, Kristy Choi +2
Normalizing flows model complex probability distributions using maps obtained by composing invertible layers. Special linear layers such as masked and 1x1 convolutions play a key r…
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Tri Dao, Daniel Y. Fu, Stefano Ermon +2
Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods ha…
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
Jue Wang, Binhang Yuan, Luka Rimanic +5
Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive…