Improving the Gating Mechanism of Recurrent Neural Networks
arXiv:1910.09890
Abstract
Gating mechanisms are widely used in neural network models, where they allow gradients to backpropagate more easily through depth or time. However, their saturation property introduces problems of its own. For example, in recurrent models these gates need to have outputs near 1 to propagate information over long time-delays, which requires them to operate in their saturation regime and hinders gradient-based learning of the gate mechanism. We address this problem by deriving two synergistic modifications to the standard gating mechanism that are easy to implement, introduce no additional hyperparameters, and improve learnability of the gates when they are close to saturation. We show how these changes are related to and improve on alternative recently proposed gating mechanisms such as chrono initialization and Ordered Neurons. Empirically, our simple gating mechanisms robustly improve the performance of recurrent models on a range of applications, including synthetic memorization tasks, sequential image classification, language modeling, and reinforcement learning, particularly when long-term dependencies are involved.
arXiv commentInternational Conference on Machine Learning 2020
References in corpus (24)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- LSTM: A Search Space Odyssey
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Categorical Reparameterization with Gumbel-Softmax
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Pointer Sentinel Mixture Models
- Learning to Execute
- A Clockwork RNN
- Hierarchical Multiscale Recurrent Neural Networks
- Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
- Control of Memory, Active Perception, and Action in Minecraft
- Unsupervised Predictive Memory in a Goal-Directed Agent
- Trellis Networks for Sequence Modeling
- Learning Longer-term Dependencies in RNNs with Auxiliary Losses
- Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
- The unreasonable effectiveness of the forget gate
- Single Headed Attention RNN: Stop Thinking With Your Head
- Recurrent Orthogonal Networks and Long-Memory Tasks
- Memory Augmented Neural Networks with Wormhole Connections
- Towards Binary-Valued Gates for Robust LSTM Training
- Fast Parametric Learning with Activation Memorization
- Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies