Dense Uncertainty Estimation
arXiv:2110.06427
Abstract
Deep neural networks can be roughly divided into deterministic neural networks and stochastic neural networks.The former is usually trained to achieve a mapping from input space to output space via maximum likelihood estimation for the weights, which leads to deterministic predictions during testing. In this way, a specific weights set is estimated while ignoring any uncertainty that may occur in the proper weight space. The latter introduces randomness into the framework, either by assuming a prior distribution over model parameters (i.e. Bayesian Neural Networks) or including latent variables (i.e. generative models) to explore the contribution of latent variables for model predictions, leading to stochastic predictions during testing. Different from the former that achieves point estimation, the latter aims to estimate the prediction distribution, making it possible to estimate uncertainty, representing model ignorance about its predictions. We claim that conventional deterministic neural network based dense prediction tasks are prone to overfitting, leading to over-confident predictions, which is undesirable for decision making. In this paper, we investigate stochastic neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. Specifically, we work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework. Due to the close connection between uncertainty estimation and model calibration, we also introduce how uncertainty estimation can be used for deep model calibration to achieve well-calibrated models, namely dense model calibration. Code and data are available at https://github.com/JingZhang617/UncertaintyEstimation.
Technical Report
References in corpus (22)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Anabranch Network for Camouflaged Object Segmentation
- Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
- Bayesian Active Learning for Classification and Preference Learning
- High Quality Monocular Depth Estimation via Transfer Learning
- Understanding Measures of Uncertainty for Adversarial Example Detection
- A Theory of Generative ConvNet
- Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
- Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
- Snapshot Ensembles: Train 1, get M for free
- Deblurring Face Images using Uncertainty Guided Multi-Stream Semantic Networks
- Alternating Back-Propagation for Generator Network
- Open Category Detection with PAC Guarantees
- Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction
- Accurate Uncertainty Estimation and Decomposition in Ensemble Learning
- Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
- Improving model calibration with accuracy versus uncertainty optimization
- Uncertainty Quantification with Generative Models
- Adaptive Semantic Segmentation with a Strategic Curriculum of Proxy Labels