On the Brittleness of Bayesian Inference

Abstract

With the advent of high-performance computing, Bayesian methods are increasingly popular tools for the quantification of uncertainty throughout science and industry. Since these methods impact the making of sometimes critical decisions in increasingly complicated contexts, the sensitivity of their posterior conclusions with respect to the underlying models and prior beliefs is a pressing question for which there currently exist positive and negative results. We report new results suggesting that, although Bayesian methods are robust when the number of possible outcomes is finite or when only a finite number of marginals of the data-generating distribution are unknown, they could be generically brittle when applied to continuous systems (and their discretizations) with finite information on the data-generating distribution. If closeness is defined in terms of the total variation metric or the matching of a finite system of generalized moments, then (1) two practitioners who use arbitrarily close models and observe the same (possibly arbitrarily large amount of) data may reach opposite conclusions; and (2) any given prior and model can be slightly perturbed to achieve any desired posterior conclusions. The mechanism causing brittlenss/robustness suggests that learning and robustness are antagonistic requirements and raises the question of a missing stability condition for using Bayesian Inference in a continuous world under finite information.

20 pages, 2 figures. To appear in SIAM Review (Research Spotlights). arXiv admin note: text overlap with arXiv:1304.6772

References in corpus (6)

• Rates of contraction of posterior distributions based on Gaussian process priors
• Misspecification in infinite-dimensional Bayesian statistics
• On the Bernstein-von Mises phenomenon for nonparametric Bayes procedures
• Nonparametric Bernstein-von Mises theorems in Gaussian white noise
• A Counter Example to Theorems of Cox and Fine
• Brittleness of Bayesian Inference Under Finite Information in a Continuous World

Cited by in corpus (13)

• Kernel Flows: from learning kernels from data into the abyss
• Bayesian Probabilistic Numerical Methods
• Learning dynamical systems from data: a simple cross-validation perspective
• Gamblets for opening the complexity-bottleneck of implicit schemes for hyperbolic and parabolic ODEs/PDEs with rough coefficients
• On the Local Lipschitz Stability of Bayesian Inverse Problems
• Model-Free Data-Driven Inference in Computational Mechanics
• Bayesian parameter identification in Cahn-Hilliard models for biological growth
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
• Methodological reconstruction of historical seismic events from anecdotal accounts of destructive tsunamis: a case study for the great 1852 Banda arc mega-thrust earthquake and tsunami
• Towards Machine Wald
• Probabilistic learning inference of boundary value problem with uncertainties based on Kullback-Leibler divergence under implicit constraints
• Multiclass classification utilising an estimated algorithmic probability prior
• Variance-based sensitivity of Bayesian inverse problems to the prior distribution