Learning Longer-term Dependencies in RNNs with Auxiliary Losses

Abstract

Despite recent advances in training recurrent neural networks (RNNs), capturing long-term dependencies in sequences remains a fundamental challenge. Most approaches use backpropagation through time (BPTT), which is difficult to scale to very long sequences. This paper proposes a simple method that improves the ability to capture long term dependencies in RNNs by adding an unsupervised auxiliary loss to the original objective. This auxiliary loss forces RNNs to either reconstruct previous events or predict next events in a sequence, making truncated backpropagation feasible for long sequences and also improving full BPTT. We evaluate our method on a variety of settings, including pixel-by-pixel image classification with sequence lengths up to 16\,000, and a real document classification benchmark. Our results highlight good performance and resource efficiency of this approach over competitive baselines, including other recurrent models and a comparable sized Transformer. Further analyses reveal beneficial effects of the auxiliary loss on optimization and regularization, as well as extreme cases where there is little to no backpropagation.

arXiv commentICML 2018

References in corpus (13)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Deep Residual Learning for Image Recognition
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Effective Approaches to Attention-based Neural Machine Translation
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Training Deep Nets with Sublinear Memory Cost
• Towards Biologically Plausible Deep Learning
• A Clockwork RNN
• Learning Longer Memory in Recurrent Neural Networks
• Neural Speed Reading via Skim-RNN
• On Multiplicative Integration with Recurrent Neural Networks
• Recurrent Orthogonal Networks and Long-Memory Tasks
• Unsupervised Pretraining for Sequence to Sequence Learning

Cited by in corpus (19)

• Deep Learning based Recommender System: A Survey and New Perspectives
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• Improving the Gating Mechanism of Recurrent Neural Networks
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Dynamical Isometry and a Mean Field Theory of LSTMs and GRUs
• General Value Function Networks
• Calibration, Entropy Rates, and Memory in Language Models
• CKConv: Continuous Kernel Convolution For Sequential Data
• Relaxed Softmax for learning from Positive and Unlabeled data
• A Cross-Domain Transferable Neural Coherence Model
• Compositional Generalization with Tree Stack Memory Units
• Explainable Deep Relational Networks for Predicting Compound-Protein Affinities and Contacts
• Learning Transition Models with Time-delayed Causal Relations
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• Image-based Natural Language Understanding Using 2D Convolutional Neural Networks
• A Machine Learning Model for Predicting Progressive Crack Extension based on DCPD Fatigue Data
• Extremal learning: extremizing the output of a neural network in regression problems