Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling

Abstract

Recurrent neural networks (RNNs) have shown promising performance for language modeling. However, traditional training of RNNs using back-propagation through time often suffers from overfitting. One reason for this is that stochastic optimization (used for large training sets) does not provide good estimates of model uncertainty. This paper leverages recent advances in stochastic gradient Markov Chain Monte Carlo (also appropriate for large training sets) to learn weight uncertainty in RNNs. It yields a principled Bayesian learning algorithm, adding gradient noise during training (enhancing exploration of the model-parameter space) and model averaging when testing. Extensive experiments on various RNN models and across a broad range of applications demonstrate the superiority of the proposed approach over stochastic optimization.

arXiv commentAccepted to ACL 2017

References in corpus (20)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Improving neural networks by preventing co-adaptation of feature detectors
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Weight Uncertainty in Neural Networks
• Visualizing and Understanding Recurrent Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Stochastic Gradient Hamiltonian Monte Carlo
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Bayesian Dark Knowledge
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Sequence-Level Knowledge Distillation
• Recurrent Dropout without Memory Loss
• Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization
• On Fast Dropout and its Applicability to Recurrent Networks
• Regularization and nonlinearities for neural language models: when are they needed?

Cited by in corpus (5)

• Bayesian Recurrent Neural Networks
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Multimodal Word Distributions
• Comparison of Deep Neural Networks and Deep Hierarchical Models for Spatio-Temporal Data
• Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning