Neural Variational Gradient Descent

Abstract

Particle-based approximate Bayesian inference approaches such as Stein Variational Gradient Descent (SVGD) combine the flexibility and convergence guarantees of sampling methods with the computational benefits of variational inference. In practice, SVGD relies on the choice of an appropriate kernel function, which impacts its ability to model the target distribution -- a challenging problem with only heuristic solutions. We propose Neural Variational Gradient Descent (NVGD), which is based on parameterizing the witness function of the Stein discrepancy by a deep neural network whose parameters are learned in parallel to the inference, mitigating the necessity to make any kernel choices whatsoever. We empirically evaluate our method on popular synthetic inference problems, real-world Bayesian linear regression, and Bayesian neural network inference.

References in corpus (20)

• Stein Variational Gradient Descent as Gradient Flow
• What Are Bayesian Neural Network Posteriors Really Like?
• Message Passing Stein Variational Gradient Descent
• A Stein variational Newton method
• Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
• Rapid Convergence of the Unadjusted Langevin Algorithm: Isoperimetry Suffices
• Stochastic Gradient MCMC with Repulsive Forces
• Bayesian Neural Network Priors Revisited
• BNNpriors: A library for Bayesian neural network inference with different prior distributions
• Projected Stein Variational Gradient Descent
• Repulsive Deep Ensembles are Bayesian
• Stein Neural Sampler
• Function Space Particle Optimization for Bayesian Neural Networks
• Sliced Kernelized Stein Discrepancy
• Annealed Stein Variational Gradient Descent
• Stochastic Stein Discrepancies
• Stein Self-Repulsive Dynamics: Benefits From Past Samples
• On Stein Variational Neural Network Ensembles
• Kernel Stein Generative Modeling
• Applying SVGD to Bayesian Neural Networks for Cyclical Time-Series Prediction and Inference