Being a Bit Frequentist Improves Bayesian Neural Networks

Abstract

Despite their compelling theoretical properties, Bayesian neural networks (BNNs) tend to perform worse than frequentist methods in classification-based uncertainty quantification (UQ) tasks such as out-of-distribution (OOD) detection. In this paper, based on empirical findings in prior works, we hypothesize that this issue is because even recent Bayesian methods have never considered OOD data in their training processes, even though this "OOD training" technique is an integral part of state-of-the-art frequentist UQ methods. To validate this, we treat OOD data as a first-class citizen in BNN training by exploring four different ways of incorporating OOD data into Bayesian inference. We show in extensive experiments that OOD-trained BNNs are competitive to recent frequentist baselines. This work thus provides strong baselines for future work in Bayesian UQ.

arXiv commentAISTATS 2022

References in corpus (12)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Evidential Deep Learning to Quantify Classification Uncertainty
• Deep Anomaly Detection with Outlier Exposure
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Practical Deep Learning with Bayesian Principles
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Certifiably Adversarially Robust Detection of Out-of-Distribution Data
• An Infinite-Feature Extension for Bayesian ReLU Nets That Fixes Their Asymptotic Overconfidence