463 citations · 1.5k across the 29 of their papers we have counts for
6 papers · 1 filter
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
Tri Dao, Beidi Chen, Kaizhao Liang +4
Overparameterized neural networks generalize well but are expensive to train. Ideally, one would like to reduce their computational cost while retaining their generalization benefi…
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
Beidi Chen, Tri Dao, Eric Winsor +3
Recent advances in efficient Transformers have exploited either the sparsity or low-rank properties of attention matrices to reduce the computational and memory bottlenecks of mode…
Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
Albert Gu, Isys Johnson, Karan Goel +4
Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with uniqu…
Knowledge Distillation as Semiparametric Inference
Tri Dao, Govinda M Kamath, Vasilis Syrgkanis +1
A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, t…
Rethinking Neural Operations for Diverse Tasks
Nicholas Roberts, Mikhail Khodak, Tri Dao +3
An important goal of AutoML is to automate-away the design of neural networks on new tasks in under-explored domains. Motivated by this goal, we study the problem of enabling users…
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps
Tri Dao, Nimit S. Sohoni, Albert Gu +5
Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference…