500 citations · 1.4k across the 23 of their papers we have counts for
24 papers
Resurrecting Recurrent Neural Networks for Long Sequences
Antonio Orvieto, Samuel L Smith, Albert Gu +4
Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perfo…
Structured State Space Models for In-Context Reinforcement Learning
Chris Lu, Yannick Schroecker, Albert Gu +4
Structured state space sequence (S4) models have recently achieved state-of-the-art performance on long-range sequence modeling tasks. These models also have fast inference speeds…
Modelling Long Range Dependencies in D: From Task-Specific to a General Purpose CNN
David M. Knigge, David W. Romero, Albert Gu +5
Performant Convolutional Neural Network (CNN) architectures must be tailored to specific tasks in order to consider the length, resolution, and dimensionality of the input data. In…
Pretraining Without Attention
Junxiong Wang, Jing Nathan Yan, Albert Gu +1
Transformers have been essential to pretraining success in NLP. While other architectures have been used, downstream accuracy is either significantly worse, or requires attention l…
S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spaces
Eric Nguyen, Karan Goel, Albert Gu +5
Visual data such as images and videos are typically modeled as discretizations of inherently continuous, multidimensional signals. Existing continuous-signal models attempt to expl…
Towards a General Purpose CNN for Long Range Dependencies in D
David W. Romero, David M. Knigge, Albert Gu +4
The use of Convolutional Neural Networks (CNNs) is widespread in Deep Learning due to a range of desirable model properties which result in an efficient and effective machine learn…