An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence
arXiv:2010.02709
Abstract
A Bayesian treatment can mitigate overconfidence in ReLU nets around the training data. But far away from them, ReLU Bayesian neural networks (BNNs) can still underestimate uncertainty and thus be asymptotically overconfident. This issue arises since the output variance of a BNN with finitely many features is quadratic in the distance from the data region. Meanwhile, Bayesian linear models with ReLU features converge, in the infinite-width limit, to a particular Gaussian process (GP) with a variance that grows cubically so that no asymptotic overconfidence can occur. While this may seem of mostly theoretical interest, in this work, we show that it can be used in practice to the benefit of BNNs. We extend finite ReLU BNNs with infinite ReLU features via the GP and show that the resulting model is asymptotically maximally uncertain far away from the data while the BNNs' predictive power is unaffected near the data. Although the resulting model approximates a full GP posterior, thanks to its structure, it can be applied \emph{post-hoc} to any pre-trained ReLU BNN at a low cost.
arXiv commentNeurIPS 2021
References in corpus (11)
- On Calibration of Modern Neural Networks
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Deep Neural Networks as Gaussian Processes
- Deep Anomaly Detection with Outlier Exposure
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- Scalable Variational Gaussian Process Classification
- Uncertainty Estimation Using a Single Deep Deterministic Neural Network
- Functional Variational Bayesian Neural Networks
- Stochastic Variational Deep Kernel Learning