Bayesian Recurrent Neural Networks

Abstract

In this work we explore a straightforward variational Bayes scheme for Recurrent Neural Networks. Firstly, we show that a simple adaptation of truncated backpropagation through time can yield good quality uncertainty estimates and superior regularisation at only a small extra computational cost during training, also reducing the amount of parameters by 80\%. Secondly, we demonstrate how a novel kind of posterior approximation yields further improvements to the performance of Bayesian RNNs. We incorporate local gradient information into the approximate posterior to sharpen it around the current batch statistics. We show how this technique is not exclusive to recurrent neural networks and can be applied more widely to train Bayesian neural networks. We also empirically demonstrate how Bayesian RNNs are superior to traditional RNNs on a language modelling benchmark and an image captioning task, as well as showing how each of these methods improve our model over a variety of other schemes for training them. We also introduce a new benchmark for studying uncertainty for language models so future methods can be easily compared.

12th Women in Machine Learning Workshop (WiML 2017), co-located with the 31st Conference on Neural Information Processing Systems (NeurIPS 2017), Long Beach, CA, USA

References in corpus (9)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Recurrent Neural Network Regularization
• Weight Uncertainty in Neural Networks
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Pointer Sentinel Mixture Models
• Learning Stochastic Recurrent Networks
• Variational Recurrent Auto-Encoders
• Stochastic Backpropagation through Mixture Density Distributions

Cited by in corpus (51)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• An Introduction to Variational Autoencoders
• Noisy Networks for Exploration
• Deep and Confident Prediction for Time Series at Uber
• SuperNNova: an open-source framework for Bayesian, Neural Network based supernova classification
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Fault Detection and Identification using Bayesian Recurrent Neural Networks
• Variational Federated Multi-Task Learning
• Deep learning algorithm for data-driven simulation of noisy dynamical system
• IG-RL: Inductive Graph Reinforcement Learning for Massive-Scale Traffic Signal Control
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Bayesian Layers: A Module for Neural Network Uncertainty
• Survey on Machine Learning for Traffic-Driven Service Provisioning in Optical Networks
• Dynamic Evaluation of Neural Sequence Models
• Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference
• Estimating Uncertainty in Neural Networks for Cardiac MRI Segmentation: A Benchmark Study
• Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
• Detection of Gravitational Waves Using Bayesian Neural Networks
• The Dark Energy Survey 5-year photometrically identified Type Ia Supernovae
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• Probabilistic Model-Agnostic Meta-Learning
• Uncertainty on Asynchronous Time Event Prediction
• Multimodal Word Distributions
• AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning
• Bayesian LSTMs in medicine
• Actively Learning what makes a Discrete Sequence Valid
• Automatic structured variational inference
• Bayesian regression explains how human participants handle parameter uncertainty
• Bayesian Methods for Semi-supervised Text Annotation
• Single- to multi-fidelity history-dependent learning with uncertainty quantification and disentanglement: application to data-driven constitutive modeling
• Stein Variational Online Changepoint Detection with Applications to Hawkes Processes and Neural Networks
• Particle Filter Recurrent Neural Networks
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
• B-SCST: Bayesian Self-Critical Sequence Training for Image Captioning
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• LSTM for Model-Based Anomaly Detection in Cyber-Physical Systems
• Automatic variational inference with cascading flows
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• BEDS-Bench: Behavior of EHR-models under Distributional Shift--A Benchmark
• Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
• Individual predictions matter: Assessing the effect of data ordering in training fine-tuned CNNs for medical imaging
• Bayesian Recurrent Framework for Missing Data Imputation and Prediction with Clinical Time Series
• Sampling-free Uncertainty Estimation in Gated Recurrent Units with Exponential Families
• To Bayes or Not To Bayes? That's no longer the question!
• Probabilistic modeling of lake surface water temperature using a Bayesian spatio-temporal graph convolutional neural network
• Where Do Human Heuristics Come From?
• Uncertainty Set Prediction of Aggregated Wind Power Generation based on Bayesian LSTM and Spatio-Temporal Analysis
• Modeling continuous-time stochastic processes using $\mathcal{N}$-Curve mixtures
• Uncertainty based Class Activation Maps for Visual Question Answering