Distributed Computation for Marginal Likelihood based Model Choice

Abstract

We propose a general method for distributed Bayesian model choice, using the marginal likelihood, where a data set is split in non-overlapping subsets. These subsets are only accessed locally by individual workers and no data is shared between the workers. We approximate the model evidence for the full data set through Monte Carlo sampling from the posterior on every subset generating a model evidence per subset. The results are combined using a novel approach which corrects for the splitting using summary statistics of the generated samples. Our divide-and-conquer approach enables Bayesian model choice in the large data setting, exploiting all available information but limiting communication between workers. We derive theoretical error bounds that quantify the resulting trade-off between computational gain and loss in precision. The embarrassingly parallel nature yields important speed-ups when used on massive data sets as illustrated by our real world experiments. In addition, we show how the suggested approach can be extended to model choice within a reversible jump setting that explores multiple feature combinations within one run.

Published in Baysian Analysis: https://projecteuclid.org/journals/bayesian-analysis/advance-publication/Distributed-Computation-for-Marginal-Likelihood-based-Model-Choice/10.1214/22-BA1321.full

References in corpus (28)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Stochastic Gradient Hamiltonian Monte Carlo
• Speeding Up MCMC by Efficient Data Subsampling
• Bayesian Nonparametric Federated Learning of Neural Networks
• On Markov chain Monte Carlo methods for tall data
• On Russian Roulette Estimates for Bayesian Inference with Doubly-Intractable Likelihoods
• Parallelizing MCMC via Weierstrass Sampler
• Bayesian Evidence and Model Selection
• Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling
• Bayesian model search and multilevel inference for SNP association studies
• Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data
• From EM to Data Augmentation: The Emergence of MCMC Bayesian Computation in the 1980s
• Better together? Statistical learning in models made of modules
• Scalable Bayes via Barycenter in Wasserstein Space
• Unbiased Markov chain Monte Carlo with couplings
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• An asymptotic analysis of distributed nonparametric methods
• Joining and splitting models with Markov melding
• Variational consensus Monte Carlo
• Unbiased estimation of log normalizing constants with applications to Bayesian cross-validation
• Hamiltonian Monte Carlo with Energy Conserving Subsampling
• FedNL: Making Newton-Type Methods Applicable to Federated Learning
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• Robust and Parallel Bayesian Model Selection
• Divide and conquer in ABC: Expectation-Progagation algorithms for likelihood-free inference
• Bounding distributional errors via density ratios
• A deterministic and computable Bernstein-von Mises theorem