A Parsimonious Tour of Bayesian Model Uncertainty

Abstract

Modern statistical software and machine learning libraries are enabling semi-automated statistical inference. Within this context, it appears easier and easier to try and fit many models to the data at hand, reversing thereby the Fisherian way of conducting science by collecting data after the scientific hypothesis (and hence the model) has been determined. The renewed goal of the statistician becomes to help the practitioner choose within such large and heterogeneous families of models, a task known as model selection. The Bayesian paradigm offers a systematized way of assessing this problem. This approach, launched by Harold Jeffreys in his 1935 book Theory of Probability, has witnessed a remarkable evolution in the last decades, that has brought about several new theoretical and methodological advances. Some of these recent developments are the focus of this survey, which tries to present a unifying perspective on work carried out by different communities. In particular, we focus on non-asymptotic out-of-sample performance of Bayesian model selection and averaging techniques, and draw connections with penalized maximum likelihood. We also describe recent extensions to wider classes of probabilistic frameworks including high-dimensional, unidentifiable, or likelihood-free models.

References in corpus (20)

• Expectation Propagation for approximate Bayesian inference
• A Widely Applicable Bayesian Information Criterion
• Comparison of Bayesian predictive methods for model selection
• Bayesian variable selection with shrinking and diffusing priors
• The discriminative functional mixture model for a comparative analysis of bike sharing systems
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Yes, but Did It Work?: Evaluating Variational Inference
• Bayesian model selection without evidences: application to the dark energy equation-of-state
• Variational Inference using Implicit Distributions
• Testing hypotheses via a mixture estimation model
• Conjugate Bayes for probit regression via unified skew-normal distributions
• Importance sampling methods for Bayesian discrimination between embedded models
• Learning Invariances using the Marginal Likelihood
• Bayes factor consistency
• Bayesian Variable Selection for Globally Sparse Probabilistic PCA
• Marginal Likelihood Integrals for Mixtures of Independence Models
• A Variational Algorithm for Bayesian Variable Selection
• A Class of Conjugate Priors for Multinomial Probit Models which Includes the Multivariate Normal One
• A review on statistical inference methods for discrete Markov random fields
• When are Bayesian model probabilities overconfident?

Cited by in corpus (5)

• $\mathtt{bajes}$: Bayesian inference of multimessenger astrophysical data, methods and application to gravitational-waves
• The Unreasonable Effectiveness of Deep Evidential Regression
• Towards calibrated and scalable uncertainty representations for neural networks
• Theoretical characterization of uncertainty in high-dimensional linear classification
• Multivariate Deep Evidential Regression