Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
arXiv:2111.03577
Abstract
Deep neural networks are prone to overconfident predictions on outliers. Bayesian neural networks and deep ensembles have both been shown to mitigate this problem to some extent. In this work, we aim to combine the benefits of the two approaches by proposing to predict with a Gaussian mixture model posterior that consists of a weighted sum of Laplace approximations of independently trained deep neural networks. The method can be used post hoc with any set of pre-trained networks and only requires a small computational and memory overhead compared to regular ensembles. We theoretically validate that our approach mitigates overconfidence "far away" from the training data and empirically compare against state-of-the-art baselines on standard uncertainty quantification benchmarks.
arXiv commentBayesian Deep Learning Workshop, NeurIPS 2021
References in corpus (18)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Wide Residual Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Weight Uncertainty in Neural Networks
- Scalable Bayesian Optimization Using Deep Neural Networks
- Deep Anomaly Detection with Outlier Exposure
- Deep Ensembles: A Loss Landscape Perspective
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
- Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
- A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
- Uncertainty Quantification and Deep Ensembles
- 'In-Between' Uncertainty in Bayesian Neural Networks
- On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
- On Power Laws in Deep Ensembles
- Benchmarking the Neural Linear Model for Regression
- Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
- Combining Ensembles and Data Augmentation can Harm your Calibration