Memory-Efficient Backpropagation Through Time

Abstract

We propose a novel approach to reduce memory consumption of the backpropagation through time (BPTT) algorithm when training recurrent neural networks (RNNs). Our approach uses dynamic programming to balance a trade-off between caching of intermediate results and recomputation. The algorithm is capable of tightly fitting within almost any user-set memory budget while finding an optimal execution policy minimizing the computational cost. Computational devices have limited memory capacity and maximizing a computational performance given a fixed memory budget is a practical use-case. We provide asymptotic computational upper bounds for various regimes. The algorithm is particularly effective for long sequences. For sequences of length 1000, our algorithm saves 95\% of memory usage while using only one third more time per iteration than the standard BPTT.

References in corpus (3)

• DRAW: A Recurrent Neural Network For Image Generation
• Training Deep Nets with Sublinear Memory Cost
• Deep Attention Recurrent Q-Network

Cited by in corpus (13)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Quantum Machine Learning: from physics to software engineering
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Efficient Scaling of Dynamic Graph Neural Networks
• Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
• Extensions and Limitations of the Neural GPU
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• VQ-DRAW: A Sequential Discrete VAE
• OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation
• Meta-Learning a Dynamical Language Model
• Hydra: A System for Large Multi-Model Deep Learning