Stein Variational Gradient Descent With Matrix-Valued Kernels

Abstract

Stein variational gradient descent (SVGD) is a particle-based inference algorithm that leverages gradient information for efficient approximate inference. In this work, we enhance SVGD by leveraging preconditioning matrices, such as the Hessian and Fisher information matrix, to incorporate geometric information into SVGD updates. We achieve this by presenting a generalization of SVGD that replaces the scalar-valued kernels in vanilla SVGD with more general matrix-valued kernels. This yields a significant extension of SVGD, and more importantly, allows us to flexibly incorporate various preconditioning matrices to accelerate the exploration in the probability landscape. Empirical results show that our method outperforms vanilla SVGD and a variety of baseline approaches over a range of real-world Bayesian inference tasks.

Neural Information Processing Systems 2019

References in corpus (12)

• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• Stein Variational Gradient Descent as Gradient Flow
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
• Message Passing Stein Variational Gradient Descent
• Stein Variational Gradient Descent With Matrix-Valued Kernels
• A Stein variational Newton method
• Stein Variational Gradient Descent Without Gradient
• Stein Point Markov Chain Monte Carlo
• Riemannian Stein Variational Gradient Descent for Bayesian Inference

Cited by in corpus (20)

• Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimation
• Stein Variational Gradient Descent With Matrix-Valued Kernels
• On the geometry of Stein variational gradient descent
• An introduction to variational inference in Geophysical inverse problems
• Bayesian inference of heterogeneous epidemic models: Application to COVID-19 spread accounting for long-term care facilities
• Stein variational gradient descent with local approximations
• Particle-based Energetic Variational Inference
• Stein variational gradient descent on infinite-dimensional space and applications to statistical inverse problems
• VIP -- Variational Inversion Package with example implementations of Bayesian tomographic imaging
• Stein Self-Repulsive Dynamics: Benefits From Past Samples
• On Stein Variational Neural Network Ensembles
• Convergence of Stein Variational Gradient Descent under a Weaker Smoothness Condition
• SVGD as a kernelized Wasserstein gradient flow of the chi-squared divergence
• Entropy Regularized Motion Planning via Stein Variational Inference
• Uncertainty Quantification of Graph Convolution Neural Network Models of Evolving Processes
• Sampling with Mirrored Stein Operators
• Vector-Valued Control Variates
• Learning Manifold Implicitly via Explicit Heat-Kernel Learning
• Scalable Approximate Inference and Some Applications
• Accelerated Information Gradient flow