Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

Abstract

Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency. We introduce a simple sequence model inspired by control systems that generalizes these approaches while addressing their shortcomings. The Linear State-Space Layer (LSSL) maps a sequence $u\mapsto y$ by simply simulating a linear continuous-time state-space representation $\dot{x}=Ax+Bu,y=Cx+Du$. Theoretically, we show that LSSL models are closely related to the three aforementioned families of models and inherit their strengths. For example, they generalize convolutions to continuous-time, explain common RNN heuristics, and share features of NDEs such as time-scale adaptation. We then incorporate and generalize recent theory on continuous-time memorization to introduce a trainable subset of structured matrices $A$ that endow LSSLs with long-range memory. Empirically, stacking LSSL layers into a simple deep neural network obtains state-of-the-art results across time series benchmarks for long dependencies in sequential image classification, real-world healthcare regression tasks, and speech. On a difficult speech classification task with length-16000 sequences, LSSL outperforms prior approaches by 24 accuracy points, and even outperforms baselines that use hand-crafted features on 100x shorter sequences.

NeurIPS 2021

References in corpus (9)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Recurrent Neural Networks in the Eye of Differential Equations
• CKConv: Continuous Kernel Convolution For Sequential Data
• Liquid Time-constant Networks

Cited by in corpus (2)

• Cottention: Linear Transformers With Cosine Attention
• Before It's Too Late: A State Space Model for the Early Prediction of Misinformation and Disinformation Engagement