Learning under Model Misspecification: Applications to Variational and Ensemble methods

Abstract

Virtually any model we use in machine learning to make predictions does not perfectly represent reality. So, most of the learning happens under model misspecification. In this work, we present a novel analysis of the generalization performance of Bayesian model averaging under model misspecification and i.i.d. data using a new family of second-order PAC-Bayes bounds. This analysis shows, in simple and intuitive terms, that Bayesian model averaging provides suboptimal generalization performance when the model is misspecified. In consequence, we provide strong theoretical arguments showing that Bayesian methods are not optimal for learning predictive models, unless the model class is perfectly specified. Using novel second-order PAC-Bayes bounds, we derive a new family of Bayesian-like algorithms, which can be implemented as variational and ensemble methods. The output of these algorithms is a new posterior distribution, different from the Bayesian posterior, which induces a posterior predictive distribution with better generalization performance. Experiments with Bayesian neural networks illustrate these findings.

arXiv commentCamera-Ready Version. NeurIPS 2020. Minor changes

References in corpus (14)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Variational Inference with Normalizing Flows
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Deep Ensembles: A Loss Landscape Perspective
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Practical Deep Learning with Bayesian Principles
• Principles of Bayesian Inference using General Divergence Criteria
• Simpler PAC-Bayesian Bounds for Hostile Data
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Accurate Uncertainty Estimation and Decomposition in Ensemble Learning
• Nonparametric learning from Bayesian models with randomized objective functions
• Second Order PAC-Bayesian Bounds for the Weighted Majority Vote
• Robust Deep Gaussian Processes

Cited by in corpus (5)

• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons
• Calibrating AI Models for Few-Shot Demodulation via Conformal Prediction
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• PAC-Bayes-Chernoff bounds for unbounded losses