Overcoming the vanishing gradient problem in plain recurrent networks

Abstract

Plain recurrent networks greatly suffer from the vanishing gradient problem while Gated Neural Networks (GNNs) such as Long-short Term Memory (LSTM) and Gated Recurrent Unit (GRU) deliver promising results in many sequence learning tasks through sophisticated network designs. This paper shows how we can address this problem in a plain recurrent network by analyzing the gating mechanisms in GNNs. We propose a novel network called the Recurrent Identity Network (RIN) which allows a plain recurrent network to overcome the vanishing gradient problem while training very deep models without the use of gates. We compare this model with IRNNs and LSTMs on multiple sequence modeling benchmarks. The RINs demonstrate competitive performance and converge faster in all tasks. Notably, small RIN models produce 12%--67% higher accuracy on the Sequential and Permuted MNIST datasets and reach state-of-the-art performance on the bAbI question answering dataset.

20 pages, 12 figures

References in corpus (6)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Training Very Deep Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Capacity and Trainability in Recurrent Neural Networks
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation

Cited by in corpus (5)

• Activation Function Optimization Scheme for Image Classification
• On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis
• Shapley Interpretation and Activation in Neural Networks
• Hybrid of DiffStride and Spectral Pooling in Convolutional Neural Networks
• Mapping high-performance RNNs to in-memory neuromorphic chips