Principles of Bayesian Inference using General Divergence Criteria

Abstract

When it is acknowledged that all candidate parameterised statistical models are misspecified relative to the data generating process, the decision maker (DM) must currently concern themselves with inference for the parameter value minimising the KL-divergence between the model and the process (Walker, 2013). However, it has long been known that minimising the KL-divergence places a large weight on correctly capturing the tails of the sample distribution. As a result the DM is required to worry about the robustness of their model to tail misspecifications if they want to conduct principled inference. In this paper we alleviate these concerns for the DM. We advance recent methodological developments in general Bayesian updating (Bissiri, Holmes and Walker, 2016) to propose a statistically well principled Bayesian updating of beliefs targeting the minimisation of more general divergence criteria. We improve both the motivation and the statistical foundations of existing Bayesian minimum divergence estimation (Hooker and Vidyashankar, 2014; Ghosh and Basu, 2016), allowing the well principled Bayesian to target predictions from the model that are close to the genuine model in terms of some alternative divergence measure to the KL-divergence. Our principled formulation allows us to consider a broader range of divergences than have previously been considered. In fact we argue defining the divergence measure forms an important, subjective part of any statistical analysis, and aim to provide some decision theoretic rational for this selection. We illustrate how targeting alternative divergence measures can impact the conclusions of simple inference tasks, and discuss then how our methods might apply to more complicated, high dimensional models.

References in corpus (4)

• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• A General Framework for Updating Belief Distributions
• Minimum scoring rule inference
• General Robust Bayes Pseudo-Posterior: Exponential Convergence results with Applications

Cited by in corpus (19)

• Generalized Variational Inference: Three arguments for deriving new Posteriors
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Learning under Model Misspecification: Applications to Variational and Ensemble methods
• Finite mixture models do not reliably learn the number of components
• Generalized Posteriors in Approximate Bayesian Computation
• Robust Bayesian Regression with Synthetic Posterior
• Robust Deep Gaussian Processes
• Generalized Bayesian Likelihood-Free Inference
• Robust Bayesian Modeling of Counts with Zero inflation and Outliers: Theoretical Robustness and Efficient Computation
• Introducing prior information in Weighted Likelihood Bootstrap with applications to model misspecification
• Sampling algorithms in statistical physics: a guide for statistics and machine learning
• Frequentist Consistency of Generalized Variational Inference
• Generalized Bayesian Filtering via Sequential Monte Carlo
• Foundations of Bayesian Learning from Synthetic Data
• Robust Bayesian Inference for Discrete Outcomes with the Total Variation Distance
• On default priors for robust Bayesian estimation with divergences
• Spatially Adaptive Ensemble Learning with Calibrated Predictive Uncertainty
• On Robust Pseudo-Bayes Estimation for the Independent Non-homogeneous Set-up
• Generalised Bayes Updates with $f$-divergences through Probabilistic Classifiers