NOMU: Neural Optimization-based Model Uncertainty
arXiv:2102.13640
Abstract
We study methods for estimating model uncertainty for neural networks (NNs) in regression. To isolate the effect of model uncertainty, we focus on a noiseless setting with scarce training data. We introduce five important desiderata regarding model uncertainty that any method should satisfy. However, we find that established benchmarks often fail to reliably capture some of these desiderata, even those that are required by Bayesian theory. To address this, we introduce a new approach for capturing model uncertainty for NNs, which we call Neural Optimization-based Model Uncertainty (NOMU). The main idea of NOMU is to design a network architecture consisting of two connected sub-NNs, one for model prediction and one for model uncertainty, and to train it using a carefully-designed loss function. Importantly, our design enforces that NOMU satisfies our five desiderata. Due to its modular architecture, NOMU can provide model uncertainty for any given (previously trained) NN if given access to its training data. We evaluate NOMU in various regressions tasks and noiseless Bayesian optimization (BO) with costly evaluations. In regression, NOMU performs at least as well as state-of-the-art methods. In BO, NOMU even outperforms all considered benchmarks.
arXiv comment9 pages + appendix
References in corpus (15)
- Geometric deep learning: going beyond Euclidean data
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
- Weight Uncertainty in Neural Networks
- Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- A Tutorial on Bayesian Optimization
- Accurate Uncertainties for Deep Learning Using Calibrated Regression
- Deep Ensembles: A Loss Landscape Perspective
- High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach
- Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
- Bayesian Optimization of Combinatorial Structures
- Regression Prior Networks
- Benchmarking the Neural Linear Model for Regression
- How Implicit Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part I: the 1-D Case of Two Layers with Random First Layer