Learnable Bernoulli Dropout for Bayesian Deep Learning

Abstract

In this work, we propose learnable Bernoulli dropout (LBD), a new model-agnostic dropout scheme that considers the dropout rates as parameters jointly optimized with other model parameters. By probabilistic modeling of Bernoulli dropout, our method enables more robust prediction and uncertainty quantification in deep models. Especially, when combined with variational auto-encoders (VAEs), LBD enables flexible semi-implicit posterior representations, leading to new semi-implicit VAE~(SIVAE) models. We solve the optimization for training with respect to the dropout parameters using Augment-REINFORCE-Merge (ARM), an unbiased and low-variance gradient estimator. Our experiments on a range of tasks show the superior performance of our approach compared with other commonly used dropout schemes. Overall, LBD leads to improved accuracy and uncertainty estimates in image classification and semantic segmentation. Moreover, using SIVAE, we can achieve state-of-the-art performance on collaborative filtering for implicit feedback on several public datasets.

To appear in AISTATS 2020

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Intriguing properties of neural networks
• Categorical Reparameterization with Gumbel-Softmax
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Weight Uncertainty in Neural Networks
• Understanding deep learning requires rethinking generalization
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Variational Dropout and the Local Reparameterization Trick
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models
• Variational Gaussian Dropout is not Bayesian
• Adaptive Activity Monitoring with Uncertainty Quantification in Switching Gaussian Process Models

Cited by in corpus (8)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Notes on the Behavior of MC Dropout
• Bayesian Graph Neural Networks with Adaptive Connection Sampling
• Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimator
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Data-driven discovery of interpretable causal relations for deep learning material laws with uncertainty propagation
• A Bayesian Neural Network based on Dropout Regulation
• ARSM Gradient Estimator for Supervised Learning to Rank