Simplified Gating in Long Short-term Memory (LSTM) Recurrent Neural Networks

Abstract

The standard LSTM recurrent neural networks while very powerful in long-range dependency sequence applications have highly complex structure and relatively large (adaptive) parameters. In this work, we present empirical comparison between the standard LSTM recurrent neural network architecture and three new parameter-reduced variants obtained by eliminating combinations of the input signal, bias, and hidden unit signals from individual gating signals. The experiments on two sequence datasets show that the three new variants, called simply as LSTM1, LSTM2, and LSTM3, can achieve comparable performance to the standard LSTM model with less (adaptive) parameters.

5 pages, 4 Figures, 3 Tables. arXiv admin note: substantial text overlap with arXiv:1612.03707

References in corpus (3)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Cited by in corpus (1)

• Simplified Long Short-term Memory Recurrent Neural Networks: part III