Learnable Bernoulli Dropout for Bayesian Deep Learning
arXiv:2002.05155
Abstract
In this work, we propose learnable Bernoulli dropout (LBD), a new model-agnostic dropout scheme that considers the dropout rates as parameters jointly optimized with other model parameters. By probabilistic modeling of Bernoulli dropout, our method enables more robust prediction and uncertainty quantification in deep models. Especially, when combined with variational auto-encoders (VAEs), LBD enables flexible semi-implicit posterior representations, leading to new semi-implicit VAE~(SIVAE) models. We solve the optimization for training with respect to the dropout parameters using Augment-REINFORCE-Merge (ARM), an unbiased and low-variance gradient estimator. Our experiments on a range of tasks show the superior performance of our approach compared with other commonly used dropout schemes. Overall, LBD leads to improved accuracy and uncertainty estimates in image classification and semantic segmentation. Moreover, using SIVAE, we can achieve state-of-the-art performance on collaborative filtering for implicit feedback on several public datasets.
To appear in AISTATS 2020
References in corpus (7)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Improving neural networks by preventing co-adaptation of feature detectors
- Weight Uncertainty in Neural Networks
- Understanding deep learning requires rethinking generalization
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Variational Gaussian Dropout is not Bayesian
- Adaptive Activity Monitoring with Uncertainty Quantification in Switching Gaussian Process Models