Distributed Computation for Marginal Likelihood based Model Choice
arXiv:1910.04672 · doi:10.1214/22-BA1321
Abstract
We propose a general method for distributed Bayesian model choice, using the marginal likelihood, where a data set is split in non-overlapping subsets. These subsets are only accessed locally by individual workers and no data is shared between the workers. We approximate the model evidence for the full data set through Monte Carlo sampling from the posterior on every subset generating a model evidence per subset. The results are combined using a novel approach which corrects for the splitting using summary statistics of the generated samples. Our divide-and-conquer approach enables Bayesian model choice in the large data setting, exploiting all available information but limiting communication between workers. We derive theoretical error bounds that quantify the resulting trade-off between computational gain and loss in precision. The embarrassingly parallel nature yields important speed-ups when used on massive data sets as illustrated by our real world experiments. In addition, we show how the suggested approach can be extended to model choice within a reversible jump setting that explores multiple feature combinations within one run.
Published in Baysian Analysis: https://projecteuclid.org/journals/bayesian-analysis/advance-publication/Distributed-Computation-for-Marginal-Likelihood-based-Model-Choice/10.1214/22-BA1321.full
References in corpus (14)
- Federated Learning: Challenges, Methods, and Future Directions
- Bayesian Nonparametric Federated Learning of Neural Networks
- On Markov chain Monte Carlo methods for tall data
- Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling
- From EM to Data Augmentation: The Emergence of MCMC Bayesian Computation in the 1980s
- Better together? Statistical learning in models made of modules
- FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
- An asymptotic analysis of distributed nonparametric methods
- Variational consensus Monte Carlo
- Unbiased estimation of log normalizing constants with applications to Bayesian cross-validation
- FedNL: Making Newton-Type Methods Applicable to Federated Learning
- Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
- Bounding distributional errors via density ratios
- A deterministic and computable Bernstein-von Mises theorem