GPU-accelerated Gibbs sampling: a case study of the Horseshoe Probit model

Abstract

Gibbs sampling is a widely used Markov chain Monte Carlo (MCMC) method for numerically approximating integrals of interest in Bayesian statistics and other mathematical sciences. Many implementations of MCMC methods do not extend easily to parallel computing environments, as their inherently sequential nature incurs a large synchronization cost. In the case study illustrated by this paper, we show how to do Gibbs sampling in a fully data-parallel manner on a graphics processing unit, for a large class of exchangeable models that admit latent variable representations. Our approach takes a systems perspective, with emphasis placed on efficient use of compute hardware. We demonstrate our method on a Horseshoe Probit regression model and find that our implementation scales effectively to thousands of predictors and millions of data points simultaneously.

References in corpus (9)

• One weird trick for parallelizing convolutional neural networks
• Simulation of truncated normal variables
• A simple sampler for the horseshoe estimator
• On the utility of graphics cards to perform massively parallel simulation of advanced Monte Carlo methods
• Random number generators for massively parallel simulations on GPU
• Asynchronous Gibbs Sampling
• Pólya Urn Latent Dirichlet Allocation: a doubly sparse massively parallel sampler
• Fast Parallel SAME Gibbs Sampling on General Discrete Bayesian Networks
• Sparse Partially Collapsed MCMC for Parallel Inference in Topic Models

Cited by in corpus (14)

• A Survey of Monte Carlo Methods for Parameter Estimation
• Asynchronous Gibbs Sampling
• Probabilistic Circuits for Autonomous Learning: A simulation study
• Pólya Urn Latent Dirichlet Allocation: a doubly sparse massively parallel sampler
• Group Inverse-Gamma Gamma Shrinkage for Sparse Regression with Block-Correlated Predictors
• Vector operations for accelerating expensive Bayesian computations -- a tutorial guide
• Bayesian Spatial Binary Regression for Label Fusion in Structural Neuroimaging
• Massive Parallelization of Massive Sample-size Survival Analysis
• Horseshoe Regularization for Machine Learning in Complex and Deep Models
• Divide-and-Conquer MCMC for Multivariate Binary Data
• Variational Inference for Shrinkage Priors: The R package vir
• Accelerating Markov Random Field Inference with Uncertainty Quantification
• Techniques for proving Asynchronous Convergence results for Markov Chain Monte Carlo methods
• Analysis of Large Heterogeneous Repairable System Reliability Data with Static System Attributes and Dynamic Sensor Measurement in Big Data Environment