Learning Longer-term Dependencies via Grouped Distributor Unit

Abstract

Learning long-term dependencies still remains difficult for recurrent neural networks (RNNs) despite their success in sequence modeling recently. In this paper, we propose a novel gated RNN structure, which contains only one gate. Hidden states in the proposed grouped distributor unit (GDU) are partitioned into groups. For each group, the proportion of memory to be overwritten in each state transition is limited to a constant and is adaptively distributed to each group member. In other word, every separate group has a fixed overall update rate, yet all units are allowed to have different paces. Information is therefore forced to be latched in a flexible way, which helps the model to capture long-term dependencies in data. Besides having a simpler structure, GDU is demonstrated experimentally to outperform LSTM and GRU on tasks including both pathological problems and natural data set.

References in corpus (9)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• LSTM: A Search Space Odyssey
• On the difficulty of training Recurrent Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Unitary Evolution Recurrent Neural Networks
• Capacity and Trainability in Recurrent Neural Networks
• Analyzing and Exploiting NARX Recurrent Neural Networks for Long-Term Dependencies