Reversible Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) provide state-of-the-art performance in processing sequential data but are memory intensive to train, limiting the flexibility of RNN models which can be trained. Reversible RNNs---RNNs for which the hidden-to-hidden transition can be reversed---offer a path to reduce the memory requirements of training, as hidden states need not be stored and instead can be recomputed during backpropagation. We first show that perfectly reversible RNNs, which require no storage of the hidden activations, are fundamentally limited because they cannot forget information from their hidden state. We then provide a scheme for storing a small number of bits in order to allow perfect reversal with forgetting. Our method achieves comparable performance to traditional models while reducing the activation memory cost by a factor of 10--15. We extend our technique to attention-based sequence-to-sequence models, where it maintains performance while reducing activation memory cost by a factor of 5--10 in the encoder, and a factor of 10--15 in the decoder.

arXiv commentPublished as a conference paper at NIPS 2018

Cited by in corpus (14)

• Hamiltonian Neural Networks
• Solving inverse problems using conditional invertible neural networks
• Forecasting Sequential Data using Consistent Koopman Autoencoders
• Neural Spline Flows
• Understanding and Mitigating Exploding Inverses in Invertible Neural Networks
• UnICORNN: A recurrent model for learning very long time dependencies
• MintNet: Building Invertible Neural Networks with Masked Convolutions
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• On the Variance of Unbiased Online Recurrent Optimization
• Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
• Equilibrated Recurrent Neural Network: Neuronal Time-Delayed Self-Feedback Improves Accuracy and Stability
• Differentiate Everything with a Reversible Embeded Domain-Specific Language
• General Invertible Transformations for Flow-based Generative Modeling
• A Survey on Green Deep Learning