HiPPO: Recurrent Memory with Optimal Polynomial Projections
arXiv:2008.07669
Abstract
A central problem in learning from sequential data is representing cumulative history in an incremental fashion as more data is processed. We introduce a general framework (HiPPO) for the online compression of continuous signals and discrete time series by projection onto polynomial bases. Given a measure that specifies the importance of each time step in the past, HiPPO produces an optimal solution to a natural online function approximation problem. As special cases, our framework yields a short derivation of the recent Legendre Memory Unit (LMU) from first principles, and generalizes the ubiquitous gating mechanism of recurrent neural networks such as GRUs. This formal framework yields a new memory update mechanism (HiPPO-LegS) that scales through time to remember all history, avoiding priors on the timescale. HiPPO-LegS enjoys the theoretical benefits of timescale robustness, fast updates, and bounded gradients. By incorporating the memory dynamics into recurrent neural networks, HiPPO RNNs can empirically capture complex temporal dependencies. On the benchmark permuted MNIST dataset, HiPPO-LegS sets a new state-of-the-art accuracy of 98.3%. Finally, on a novel trajectory classification task testing robustness to out-of-distribution timescales and missing data, HiPPO-LegS outperforms RNN and neural ODE baselines by 25-40% accuracy.
References in corpus (16)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- LSTM: A Search Space Odyssey
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- On the difficulty of training Recurrent Neural Networks
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Generating Long Sequences with Sparse Transformers
- Reformer: The Efficient Transformer
- Pay Less Attention with Lightweight and Dynamic Convolutions
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- How to train your neural ODE: the world of Jacobian and kinetic regularization
- The UEA multivariate time series classification archive, 2018
- Trellis Networks for Sequence Modeling
- Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
- Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations
- SNODE: Spectral Discretization of Neural ODEs for System Identification
- Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps