Adaptively Truncating Backpropagation Through Time to Control Gradient Bias

Abstract

Truncated backpropagation through time (TBPTT) is a popular method for learning in recurrent neural networks (RNNs) that saves computation and memory at the cost of bias by truncating backpropagation after a fixed number of lags. In practice, choosing the optimal truncation length is difficult: TBPTT will not converge if the truncation length is too small, or will converge slowly if it is too large. We propose an adaptive TBPTT scheme that converts the problem from choosing a temporal lag to one of choosing a tolerable amount of gradient bias. For many realistic RNNs, the TBPTT gradients decay geometrically in expectation for large lags; under this condition, we can control the bias by varying the truncation length adaptively. For RNNs with smooth activation functions, we prove that this bias controls the convergence rate of SGD with biased gradients for our non-convex loss. Using this theory, we develop a practical method for adaptively estimating the truncation length during training. We evaluate our adaptive TBPTT method on synthetic data and language modeling tasks and find that our adaptive TBPTT ameliorates the computational pitfalls of fixed TBPTT.

References in corpus (5)

• On the difficulty of training Recurrent Neural Networks
• Pointer Sentinel Mixture Models
• Quasi-Recurrent Neural Networks
• Unbiasing Truncated Backpropagation Through Time
• Stochastic Gradient Descent with Biased but Consistent Gradient Estimators

Cited by in corpus (6)

• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Expert System Gradient Descent Style Training: Development of a Defensible Artificial Intelligence Technique
• Recurrent Quantum Neural Networks
• Prediction of the Position of External Markers Using a Recurrent Neural Network Trained With Unbiased Online Recurrent Optimization for Safe Lung Cancer Radiotherapy
• Determining Sentencing Recommendations and Patentability Using a Machine Learning Trained Expert System
• Fake News and Phishing Detection Using a Machine Learning Trained Expert System