Parallelizing MCMC via Weierstrass Sampler

Abstract

With the rapidly growing scales of statistical problems, subset based communication-free parallel MCMC methods are a promising future for large scale Bayesian analysis. In this article, we propose a new Weierstrass sampler for parallel MCMC based on independent subsets. The new sampler approximates the full data posterior samples via combining the posterior draws from independent subset MCMC chains, and thus enjoys a higher computational efficiency. We show that the approximation error for the Weierstrass sampler is bounded by some tuning parameters and provide suggestions for choice of the values. Simulation study shows the Weierstrass sampler is very competitive compared to other methods for combining MCMC chains generated for subsets, including averaging and kernel smoothing.

The original Algorithm 1 removed. Provided some theoretical justification for refinement sampling (Theorem 2). Added a new algorithm in addition to the rejection sampling for handling dimensionality curse. New simulations and graphs (with new colors and designs). A real data analysis is also provided

Cited by in corpus (51)

• Quantile Regression Under Memory Constraint
• Speeding Up MCMC by Efficient Data Subsampling
• Fusion of Probability Density Functions
• Orthogonal parallel MCMC methods for sampling and optimization
• Distributed ARIMA Models for Ultra-long Time Series
• Expectation propagation as a way of life: A framework for Bayesian inference on partitioned data
• Stochastic Variational Inference for Hidden Markov Models
• Speeding Up MCMC by Delayed Acceptance and Data Subsampling
• Scalable Bayes via Barycenter in Wasserstein Space
• Communication-Efficient Distributed Statistical Inference
• A Survey of Bayesian Statistical Approaches for Big Data
• Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
• Partitioned Variational Inference: A unified framework encompassing federated and continual learning
• DECOrrelated feature space partitioning for distributed sparse regression
• Monte Carlo Fusion
• Median Selection Subset Aggregation for Parallel Inference
• Escaping the curse of dimensionality in Bayesian model based clustering
• Bayesian computation: a perspective on the current state, and sampling backwards and forwards
• Big Learning with Bayesian Methods
• Hamiltonian Monte Carlo with Energy Conserving Subsampling
• Accelerating Metropolis-Hastings algorithms: Delayed acceptance with prefetching
• Embarrassingly Parallel Variational Inference in Nonconjugate Models
• Efficient MCMC Sampling with Dimension-Free Convergence Rate using ADMM-type Splitting
• Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server
• Distributed Computation for Marginal Likelihood based Model Choice
• Average of Recentered Parallel MCMC for Big Data
• The divide-and-conquer sequential Monte Carlo algorithm: theoretical properties and limit theorems
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• Mini-batch Metropolis-Hastings MCMC with Reversible SGLD Proposal
• Parallelising MCMC via Random Forests
• Quasi Markov Chain Monte Carlo Methods
• Subsampling MCMC - An introduction for the survey statistician
• Most Likely Optimal Subsampled Markov Chain Monte Carlo
• Parallel Markov Chain Monte Carlo via Spectral Clustering
• Consensus Monte Carlo for Random Subsets using Shared Anchors
• Multivariate spectral downscaling for PM2.5 species
• Parallelisation of a Common Changepoint Detection Method
• Scalable Bayesian Nonparametric Clustering and Classification
• Accelerating MCMC Algorithms
• A Decentralized Approach to Bayesian Learning
• Adaptive Component-wise Multiple-Try Metropolis Sampling
• Bayesian inference in hierarchical models by combining independent posteriors
• A Scalable Blocked Gibbs Sampling Algorithm For Gaussian And Poisson Regression Models
• Asymptotic properties of parallel Bayesian kernel density estimators
• Parallelizing MCMC with Machine Learning Classifier and Its Criterion Based on Kullback-Leibler Divergence
• A selective review on calibration information from similar studies based on parametric likelihood or empirical likelihood
• Parallel Markov Chain Monte Carlo for Bayesian Hierarchical Models with Big Data, in Two Stages
• Bayesian Mosaic: Parallelizable Composite Posterior
• An Algorithm for Distributed Bayesian Inference in Generalized Linear Models
• Communication-Free Parallel Supervised Topic Models
• Discussion of "Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing"