Can recurrent neural networks warp time?

Abstract

Successful recurrent models such as long short-term memories (LSTMs) and gated recurrent units (GRUs) use ad hoc gating mechanisms. Empirically these models have been found to improve the learning of medium to long term temporal dependencies and to help with vanishing gradient issues. We prove that learnable gates in a recurrent model formally provide quasi- invariance to general time transformations in the input data. We recover part of the LSTM architecture from a simple axiomatic approach. This result leads to a new way of initializing gate biases in LSTMs and GRUs. Ex- perimentally, this new chrono initialization is shown to greatly improve learning of long term dependencies, with minimal implementation effort.

References in corpus (3)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Full-Capacity Unitary Recurrent Neural Networks

Cited by in corpus (11)

• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• The unreasonable effectiveness of the forget gate
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Hyperbolic Neural Networks
• Few-Shot Generalization Across Dialogue Tasks
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
• Attention to Warp: Deep Metric Learning for Multivariate Time Series
• Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions
• T-RECS: Training for Rate-Invariant Embeddings by Controlling Speed for Action Recognition
• Reducing state updates via Gaussian-gated LSTMs