On the difficulty of training Recurrent Neural Networks

Abstract

There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.

Improved description of the exploding gradient problem and description and analysis of the vanishing gradient problem

References in corpus (3)

• Theano: new features and speed improvements
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Advances in Optimizing Recurrent Networks

Cited by in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• Sequence to Sequence Learning with Neural Networks
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Learning with Pseudo-Ensembles
• Blocks and Fuel: Frameworks for deep learning
• Self-Adaptive Hierarchical Sentence Model
• Recurrent Neural Networks with External Memory for Language Understanding
• Connectionist-Symbolic Machine Intelligence using Cellular Automata based Reservoir-Hyperdimensional Computing
• A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion
• An Empirical Study of the L2-Boost technique with Echo State Networks
• Regularizing Recurrent Networks - On Injected Noise and Norm-based Methods