Dropout as a Bayesian Approximation: Appendix

Abstract

We show that a neural network with arbitrary depth and non-linearities, with dropout applied before every weight layer, is mathematically equivalent to an approximation to a well known Bayesian model. This interpretation might offer an explanation to some of dropout's key properties, such as its robustness to over-fitting. Our interpretation allows us to reason about uncertainty in deep learning, and allows the introduction of the Bayesian machinery into existing deep learning frameworks in a principled way. This document is an appendix for the main paper "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning" by Gal and Ghahramani, 2015.

20 pages, 1 figure; ICML proceedings version

References in corpus (7)

• Improving neural networks by preventing co-adaptation of feature detectors
• Playing Atari with Deep Reinforcement Learning
• Weight Uncertainty in Neural Networks
• Variational Bayesian Inference with Stochastic Search
• Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models
• Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs
• Latent Gaussian Processes for Distribution Estimation of Multivariate Categorical Data

Cited by in corpus (18)

• Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning
• Probabilistic neural networks for fluid flow surrogate modeling and data recovery
• Qualitative Analysis of Monte Carlo Dropout
• Real-Time Detection of Anomalies in Large-Scale Transient Surveys
• Variational Inference to Measure Model Uncertainty in Deep Neural Networks
• Reconstructing the Hubble diagram of gamma-ray bursts using deep learning
• LiBRe: A Practical Bayesian Approach to Adversarial Detection
• Uncertainty-Aware Lookahead Factor Models for Quantitative Investing
• MEPG: A Minimalist Ensemble Policy Gradient Framework for Deep Reinforcement Learning
• Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference
• Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network
• Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
• Exploring Uncertainty in Conditional Multi-Modal Retrieval Systems
• Towards Principled Uncertainty Estimation for Deep Neural Networks
• On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks
• Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods
• That Label's Got Style: Handling Label Style Bias for Uncertain Image Segmentation
• DiverseNet: When One Right Answer is not Enough