Variational Inference with Holder Bounds
arXiv:2111.02947
Abstract
The recent introduction of thermodynamic integration techniques has provided a new framework for understanding and improving variational inference (VI). In this work, we present a careful analysis of the thermodynamic variational objective (TVO), bridging the gap between existing variational objectives and shedding new insights to advance the field. In particular, we elucidate how the TVO naturally connects the three key variational schemes, namely the importance-weighted VI, Renyi-VI, and MCMC-VI, which subsumes most VI objectives employed in practice. To explain the performance gap between theory and practice, we reveal how the pathological geometry of thermodynamic curves negatively affects TVO. By generalizing the integration path from the geometric mean to the weighted Holder mean, we extend the theory of TVO and identify new opportunities for improving VI. This motivates our new VI objectives, named the Holder bounds, which flatten the thermodynamic curves and promise to achieve a one-step approximation of the exact marginal log-likelihood. A comprehensive discussion on the choices of numerical estimators is provided. We present strong empirical evidence on both synthetic and real-world datasets to support our claims.
References in corpus (22)
- Auto-Encoding Variational Bayes
- Categorical Reparameterization with Gumbel-Softmax
- The Entropy Production Fluctuation Theorem and the Nonequilibrium Work Relation for Free Energy Differences
- Expectation Propagation for approximate Bayesian inference
- Variational Inference with Normalizing Flows
- Density estimation using Real NVP
- Adversarially Learned Inference
- Style Transfer from Non-Parallel Text by Cross-Alignment
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Normalizing Flows for Probabilistic Modeling and Inference
- Auxiliary Deep Generative Models
- Learning in Implicit Generative Models
- Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
- Variational inference for Monte Carlo objectives
- Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
- On integral probability metrics, ϕ-divergences and binary classification
- Sticking the Landing: Simple, Lower-Variance Gradient Estimators for Variational Inference
- Variational Boosting: Iteratively Refining Posterior Approximations
- VAE Learning via Stein Variational Gradient Descent
- Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
- All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
- Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping