Towards Binary-Valued Gates for Robust LSTM Training

Abstract

Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. It aims to use gates to control information flow (e.g., whether to skip some information or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this goal. In this paper, we propose a new way for LSTM training, which pushes the output values of the gates towards 0 or 1. By doing so, we can better control the information flow: the gates are mostly open or closed, instead of in a middle state, which makes the results more interpretable. Empirical studies show that (1) Although it seems that we restrict the model capacity, there is no performance drop: we achieve better or comparable performances due to its better generalization ability; (2) The outputs of gates are not sensitive to their inputs: we can easily compress the LSTM unit in multiple ways, e.g., low-rank approximation and low-precision approximation. The compressed models are even better than the baseline models without compression.

arXiv commentICML 2018

References in corpus (26)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• ADADELTA: An Adaptive Learning Rate Method
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Convolutional Sequence to Sequence Learning
• Recurrent Neural Network Regularization
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Sequence Level Training with Recurrent Neural Networks
• Exploring the Limits of Language Modeling
• Visualizing and Understanding Recurrent Networks
• Effective Approaches to Attention-based Neural Machine Translation
• Dual Learning for Machine Translation
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• An Actor-Critic Algorithm for Sequence Prediction
• GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
• Learning to Generate Long-term Future via Hierarchical Prediction
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Sequence-to-Sequence Learning as Beam-Search Optimization
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Massive Exploration of Neural Machine Translation Architectures
• Automatic Rule Extraction from Long Short Term Memory Networks
• Minimum Risk Training for Neural Machine Translation
• Towards Neural Phrase-based Machine Translation

Cited by in corpus (12)

• Recurrent Neural Networks: An Embedded Computing Perspective
• Improving the Gating Mechanism of Recurrent Neural Networks
• Binarized Collaborative Filtering with Distilling Graph Convolutional Networks
• National-scale electricity peak load forecasting: Traditional, machine learning, or hybrid model?
• Training Progressively Binarizing Deep Networks Using FPGAs
• Modeling and Predicting Popularity Dynamics via Deep Learning Attention Mechanism
• Tensor train decompositions on recurrent networks
• Hierarchical Context enabled Recurrent Neural Network for Recommendation
• Adversarial Learning of Label Dependency: A Novel Framework for Multi-class Classification
• Attention: to Better Stand on the Shoulders of Giants
• Learning in Gated Neural Networks
• ARMIN: Towards a More Efficient and Light-weight Recurrent Memory Network