Noisin: Unbiased Regularization for Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) are powerful models of sequential data. They have been successfully used in domains such as text and speech. However, RNNs are susceptible to overfitting; regularization is important. In this paper we develop Noisin, a new method for regularizing RNNs. Noisin injects random noise into the hidden states of the RNN and then maximizes the corresponding marginal likelihood of the data. We show how Noisin applies to any RNN and we study many different types of noise. Noisin is unbiased--it preserves the underlying RNN on average. We characterize how Noisin regularizes its RNN both theoretically and empirically. On language modeling benchmarks, Noisin improves over dropout by as much as 12.2% on the Penn Treebank and 9.4% on the Wikitext-2 dataset. We also compared the state-of-the-art language model of Yang et al. 2017, both with and without Noisin. On the Penn Treebank, the method with Noisin more quickly reaches state-of-the-art performance.

arXiv commentIn Proceedings of the International Conference on Machine Learning, 2018

References in corpus (14)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Recurrent Neural Network Regularization
• DRAW: A Recurrent Neural Network For Image Generation
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• Learning Stochastic Recurrent Networks
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Sequential Neural Models with Stochastic Layers
• Recurrent Dropout without Memory Loss
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Recurrent Batch Normalization
• Z-Forcing: Training Stochastic Recurrent Networks

Cited by in corpus (7)

• Explicit Regularisation in Gaussian Noise Injections
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• State-Regularized Recurrent Neural Networks
• Noisy Recurrent Neural Networks
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
• Medi-Care AI: Predicting Medications From Billing Codes via Robust Recurrent Neural Networks
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN