MCMC-Based Inference in the Era of Big Data: A Fundamental Analysis of the Convergence Complexity of High-Dimensional Chains

Abstract

Markov chain Monte Carlo (MCMC) lies at the core of modern Bayesian methodology, much of which would be impossible without it. Thus, the convergence properties of MCMCs have received significant attention, and in particular, proving (geometric) ergodicity is of critical interest. Trust in the ability of MCMCs to sample from modern-day high-dimensional posteriors, however, has been limited by a widespread perception that these chains typically experience serious convergence problems. In this paper, we first demonstrate that contemporary methods for obtaining convergence rates have serious limitations when the dimension grows. We then propose a framework for rigorously establishing the convergence behavior of commonly used high-dimensional MCMCs. In particular, we demonstrate theoretically the precise nature and severity of the convergence problems of popular MCMCs when implemented in high dimensions, including phase transitions in the convergence rates in various $n$ and $p$ regimes, and a universality result across an entire spectrum of models. We also show that convergence problems effectively eliminate the apparent safeguard of geometric ergodicity. We then demonstrate theoretical principles by which MCMCs can be constructed and analyzed to yield bounded geometric convergence rates even as the dimension $p$ grows without bound. Additionally, we propose a diagnostic tool for establishing convergence.

References in corpus (2)

• A General Framework for the Parametrization of Hierarchical Models
• Convergence rate of Markov chain methods for genomic motif discovery

Cited by in corpus (15)

• Complexity Results for MCMC derived from Quantitative Bounds
• Bayesian Inference for Large Scale Image Classification
• Random weighting in LASSO regression
• Towards fast machine-learning-assisted Bayesian posterior inference of microseismic event location and source mechanism
• Fast Variational Inference for Bayesian Factor Analysis in Single and Multi-Study Settings
• Optimal Scaling of Random-Walk Metropolis Algorithms on General Target Distributions
• Scalable Bayesian shrinkage and uncertainty quantification in high-dimensional regression
• On the limitations of single-step drift and minorization in Markov chain convergence analysis
• Central Limit Theorems for Markov Chains from Wasserstein Convergence Rates
• Bayesian Lasso Posterior Sampling via Parallelized Measure Transport
• Analytic solution and stationary phase approximation for the Bayesian lasso and elastic net
• Fast Markov chain Monte Carlo for high dimensional Bayesian regression models with shrinkage priors
• A Short Note on the Efficiency of Markov Chains for Bayesian Linear Regression Models with Heavy-Tailed Errors
• Multilevel linear models, Gibbs samplers and multigrid decompositions
• Bayesian inference in hierarchical models by combining independent posteriors