Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units

Abstract

Recurrent neural network (RNN) has been widely studied in sequence learning tasks, while the mainstream models (e.g., LSTM and GRU) rely on the gating mechanism (in control of how information flows between hidden states). However, the vanilla gates in RNN (e.g., the input gate in LSTM) suffer from the problem of gate undertraining, which can be caused by various factors, such as the saturating activation functions, the gate layouts (e.g., the gate number and gating functions), or even the suboptimal memory state etc.. Those may result in failures of learning gating switch roles and thus the weak performance. In this paper, we propose a new gating mechanism within general gated recurrent neural networks to handle this issue. Specifically, the proposed gates directly short connect the extracted input features to the outputs of vanilla gates, denoted as refined gates. The refining mechanism allows enhancing gradient back-propagation as well as extending the gating activation scope, which can guide RNN to reach possibly deeper minima. We verify the proposed gating mechanism on three popular types of gated RNNs including LSTM, GRU and MGU. Extensive experiments on 3 synthetic tasks, 3 language modeling tasks and 5 scene text recognition benchmarks demonstrate the effectiveness of our method.

References in corpus (20)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• LSTM: A Search Space Odyssey
• On the difficulty of training Recurrent Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• Training Very Deep Networks
• Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
• Focusing Attention: Towards Accurate Text Recognition in Natural Images
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Quasi-Recurrent Neural Networks
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• On Multiplicative Integration with Recurrent Neural Networks
• Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition
• Highway Long Short-Term Memory RNNs for Distant Speech Recognition
• Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
• Path-Normalized Optimization of Recurrent Neural Networks with ReLU Activations
• A Flexible Approach to Automated RNN Architecture Generation
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies