Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
arXiv:1902.06704
Abstract
Modelling long-term dependencies is a challenge for recurrent neural networks. This is primarily due to the fact that gradients vanish during training, as the sequence length increases. Gradients can be attenuated by transition operators and are attenuated or dropped by activation functions. Canonical architectures like LSTM alleviate this issue by skipping information through a memory mechanism. We propose a new recurrent architecture (Non-saturating Recurrent Unit; NRU) that relies on a memory mechanism but forgoes both saturating activation functions and saturating gates, in order to further alleviate vanishing gradients. In a series of synthetic and real world tasks, we demonstrate that the proposed model is the only model that performs among the top 2 models across all tasks with and without long-term dependencies, when compared against a range of other architectures.
arXiv commentIn Proceedings of AAAI 2019
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Categorical Reparameterization with Gumbel-Softmax
- Empirical Evaluation of Rectified Activations in Convolutional Network
- A Neural Conversational Model
- Layer Normalization
- Regularizing and Optimizing LSTM Language Models
- On orthogonality and learning recurrent networks with long term dependencies
- Noisy Activation Functions
- Recurrent Orthogonal Networks and Long-Memory Tasks
- Memory Augmented Neural Networks with Wormhole Connections
- Can recurrent neural networks warp time?
- Rotational Unit of Memory
Cited by in corpus (4)
- Improving the Gating Mechanism of Recurrent Neural Networks
- Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
- A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions
- Refined Gate: A Simple and Effective Gating Mechanism for Recurrent Units