Empirical Evaluation of A New Approach to Simplifying Long Short-term Memory (LSTM)
arXiv:1612.03707
Abstract
The standard LSTM, although it succeeds in the modeling long-range dependences, suffers from a highly complex structure that can be simplified through modifications to its gate units. This paper was to perform an empirical comparison between the standard LSTM and three new simplified variants that were obtained by eliminating input signal, bias and hidden unit signal from individual gates, on the tasks of modeling two sequence datasets. The experiments show that the three variants, with reduced parameters, can achieve comparable performance with the standard LSTM. Due attention should be paid to turning the learning rate to achieve high accuracies
5 pages, 5 figures
References in corpus (5)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- On the difficulty of training Recurrent Neural Networks
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Recurrent Neural Network Regularization
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units