Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data

Abstract

A common divide-and-conquer approach for Bayesian computation with big data is to partition the data, perform local inference for each piece separately, and combine the results to obtain a global posterior approximation. While being conceptually and computationally appealing, this method involves the problematic need to also split the prior for the local inferences; these weakened priors may not provide enough regularization for each separate computation, thus eliminating one of the key advantages of Bayesian methods. To resolve this dilemma while still retaining the generalizability of the underlying local inference method, we apply the idea of expectation propagation (EP) as a framework for distributed Bayesian inference. The central idea is to iteratively update approximations to the local likelihoods given the state of the other approximations and the prior. The present paper has two roles: we review the steps that are needed to keep EP algorithms numerically stable, and we suggest a general approach, inspired by EP, for approaching data partitioning problems in a way that achieves the computational benefits of parallelism while allowing each local update to make use of relevant information from the other sites. In addition, we demonstrate how the method can be applied in a hierarchical context to make use of partitioning of both data and parameters. The paper describes a general algorithmic framework, rather than a specific algorithm, and presents an example implementation for it.

Minor revision

References in corpus (7)

• Expectation Propagation for approximate Bayesian inference
• On Markov chain Monte Carlo methods for tall data
• Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
• Parallelizing MCMC via Weierstrass Sampler
• Bounding errors of Expectation-Propagation
• Adiabatic Monte Carlo
• Nested Expectation Propagation for Gaussian Process Classification with a Multinomial Probit Likelihood

Cited by in corpus (12)

• Fusion of Probability Density Functions
• Client Selection for Federated Bayesian Learning
• Federated Data Analytics: A Study on Linear Models
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Pragmatic hypotheses in the evolution of science
• BayesPerf: Minimizing Performance Monitoring Errors Using Bayesian Statistics
• Fast Scalable Image Restoration using Total Variation Priors and Expectation Propagation
• Effect of natural gamma background radiation on portal monitor radioisotope unmixing
• Distributed Computation for Marginal Likelihood based Model Choice
• Exact solution to the random sequential dynamics of a message passing algorithm
• STAR NRE: Solving supernova selection effects with set-based truncated auto-regressive neural ratio estimation
• Doubly Adaptive Importance Sampling