The unreasonable effectiveness of the forget gate

Abstract

Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-initialized biases, not only provides computational savings but outperforms the standard LSTM on multiple benchmark datasets and competes with some of the best contemporary models. Our proposed network, the JANET, achieves accuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming the standard LSTM which yields accuracies of 98.5% and 91%.

arXiv commentCorrected LSTM gradient derivations. Added link to code

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• LSTM: A Search Space Odyssey
• Deep Residual Learning for Image Recognition
• WaveNet: A Generative Model for Raw Audio
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Training Very Deep Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Sequential Neural Models with Stochastic Layers
• Dilated Recurrent Neural Networks
• Architectural Complexity Measures of Recurrent Neural Networks
• Depth-Gated LSTM
• Natural Language Understanding with Distributed Representation
• Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning
• Can recurrent neural networks warp time?

Cited by in corpus (12)

• Optimizing Speech Recognition For The Edge
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Improving the Gating Mechanism of Recurrent Neural Networks
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay
• DeltaDPD: Exploiting Dynamic Temporal Sparsity in Recurrent Neural Networks for Energy-Efficient Wideband Digital Predistortion
• Multi-timescale memory dynamics in a reinforcement learning network with attention-gated memory
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• A Dynamically Controlled Recurrent Neural Network for Modeling Dynamical Systems
• A Brief Survey of Associations Between Meta-Learning and General AI
• Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions
• A Novel Framework for Recurrent Neural Networks with Enhancing Information Processing and Transmission between Units