A Simple LSTM model for Transition-based Dependency Parsing
arXiv:1708.08959
Abstract
We present a simple LSTM-based transition-based dependency parser. Our model is composed of a single LSTM hidden layer replacing the hidden layer in the usual feed-forward network architecture. We also propose a new initialization method that uses the pre-trained weights from a feed-forward neural network to initialize our LSTM-based model. We also show that using dropout on the input layer has a positive effect on performance. Our final parser achieves a 93.06% unlabeled and 91.01% labeled attachment score on the Penn Treebank. We additionally replace LSTMs with GRUs and Elman units in our model and explore the effectiveness of our initialization method on individual gates constituting all three types of RNN units.
References in corpus (8)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Improving neural networks by preventing co-adaptation of feature detectors
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Recurrent Neural Network Regularization
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Transition-Based Dependency Parsing with Stack Long Short-Term Memory
- Learning Longer Memory in Recurrent Neural Networks
- Structured Training for Neural Network Transition-Based Parsing