Going Wider: Recurrent Neural Network With Parallel Cells
arXiv:1705.01346
Abstract
Recurrent Neural Network (RNN) has been widely applied for sequence modeling. In RNN, the hidden states at current step are full connected to those at previous step, thus the influence from less related features at previous step may potentially decrease model's learning ability. We propose a simple technique called parallel cells (PCs) to enhance the learning ability of Recurrent Neural Network (RNN). In each layer, we run multiple small RNN cells rather than one single large cell. In this paper, we evaluate PCs on 2 tasks. On language modeling task on PTB (Penn Tree Bank), our model outperforms state of art models by decreasing perplexity from 78.6 to 75.3. On Chinese-English translation task, our model increases BLEU score for 0.39 points than baseline model.
References in corpus (5)
- Improving neural networks by preventing co-adaptation of feature detectors
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Recurrent Neural Network Regularization
- Effective Approaches to Attention-based Neural Machine Translation
- A Unified Tagging Solution: Bidirectional LSTM Recurrent Neural Network with Word Embedding