Regularizing RNNs by Stabilizing Activations

Abstract

We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states' norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modeling and phoneme recognition, and outperforming weight noise and dropout. We achieve competitive performance (18.6\% PER) on the TIMIT phoneme recognition task for RNNs evaluated without beam search or an RNN transducer. With this penalty term, IRNN can achieve similar performance to LSTM on language modeling, although adding the penalty term to the LSTM results in superior performance. Our penalty term also prevents the exponential growth of IRNN's activations outside of their training horizon, allowing them to generalize to much longer sequences.

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• Recurrent Neural Network Regularization
• Theano: new features and speed improvements
• Blocks and Fuel: Frameworks for deep learning

Cited by in corpus (16)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Recent Advances in Recurrent Neural Networks
• Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
• Hierarchical Multiscale Recurrent Neural Networks
• Data Noising as Smoothing in Neural Network Language Models
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• On orthogonality and learning recurrent networks with long term dependencies
• On Multiplicative Integration with Recurrent Neural Networks
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• The Implicit and Explicit Regularization Effects of Dropout
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Shifting Mean Activation Towards Zero with Bipolar Activation Functions
• Finding Algebraic Structure of Care in Time: A Deep Learning Approach
• Fast ES-RNN: A GPU Implementation of the ES-RNN Algorithm
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
• RNN-based Online Learning: An Efficient First-Order Optimization Algorithm with a Convergence Guarantee