Message Passing Stein Variational Gradient Descent

Abstract

Stein variational gradient descent (SVGD) is a recently proposed particle-based Bayesian inference method, which has attracted a lot of interest due to its remarkable approximation ability and particle efficiency compared to traditional variational inference and Markov Chain Monte Carlo methods. However, we observed that particles of SVGD tend to collapse to modes of the target distribution, and this particle degeneracy phenomenon becomes more severe with higher dimensions. Our theoretical analysis finds out that there exists a negative correlation between the dimensionality and the repulsive force of SVGD which should be blamed for this phenomenon. We propose Message Passing SVGD (MP-SVGD) to solve this problem. By leveraging the conditional independence structure of probabilistic graphical models (PGMs), MP-SVGD converts the original high-dimensional global inference problem into a set of local ones over the Markov blanket with lower dimensions. Experimental results show its advantages of preventing vanishing repulsive force in high-dimensional space over SVGD, and its particle efficiency and approximation flexibility over other inference methods on graphical models.

arXiv commentTo appear in the Proceedings of the 35th International Conference on Machine Learning (ICML 2018)

Cited by in corpus (17)

• Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
• Stein Variational Gradient Descent With Matrix-Valued Kernels
• On the geometry of Stein variational gradient descent
• Bayesian inference of heterogeneous epidemic models: Application to COVID-19 spread accounting for long-term care facilities
• Stein Variational Model Predictive Control
• Annealed Stein Variational Gradient Descent
• Stochastic Stein Discrepancies
• On Stein Variational Neural Network Ensembles
• Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent
• Blindness of score-based methods to isolated components and mixing proportions
• Kernelized Complete Conditional Stein Discrepancy
• Stochastic Particle-Optimization Sampling and the Non-Asymptotic Convergence Theory
• Sampling with Mirrored Stein Operators
• Variational Inference MPC using Tsallis Divergence
• Stabilizing Training of Generative Adversarial Nets via Langevin Stein Variational Gradient Descent
• Relative Entropy Gradient Sampler for Unnormalized Distributions
• Neural Variational Gradient Descent