An Empirical Exploration of Skip Connections for Sequential Tagging
arXiv:1610.03167
Abstract
In this paper, we empirically explore the effects of various kinds of skip connections in stacked bidirectional LSTMs for sequential tagging. We investigate three kinds of skip connections connecting to LSTM cells: (a) skip connections to the gates, (b) skip connections to the internal states and (c) skip connections to the cell outputs. We present comprehensive experiments showing that skip connections to cell outputs outperform the remaining two. Furthermore, we observe that using gated identity functions as skip mappings works pretty well. Based on this novel skip connections, we successfully train deep stacked bidirectional LSTM models and obtain state-of-the-art results on CCG supertagging and comparable results on POS tagging.
arXiv commentAccepted at COLING 2016
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- ADADELTA: An Adaptive Learning Rate Method
- Deep Residual Learning for Image Recognition
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Grid Long Short-Term Memory
- Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
- Part-of-Speech Tagging with Bidirectional Long Short-Term Memory Recurrent Neural Network