Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
arXiv:2010.08258
Abstract
The learning and evaluation of energy-based latent variable models (EBLVMs) without any structural assumptions are highly challenging, because the true posteriors and the partition functions in such models are generally intractable. This paper presents variational estimates of the score function and its gradient with respect to the model parameters in a general EBLVM, referred to as VaES and VaGES respectively. The variational posterior is trained to minimize a certain divergence to the true model posterior and the bias in both estimates can be bounded by the divergence theoretically. With a minimal model assumption, VaES and VaGES can be applied to the kernelized Stein discrepancy (KSD) and score matching (SM)-based methods to learn EBLVMs. Besides, VaES can also be used to estimate the exact Fisher divergence between the data and general EBLVMs.
References in corpus (18)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Categorical Reparameterization with Gumbel-Softmax
- NICE: Non-linear Independent Components Estimation
- Score-Based Generative Modeling through Stochastic Differential Equations
- Generative Modeling by Estimating Gradients of the Data Distribution
- MMD GAN: Towards Deeper Understanding of Moment Matching Network
- Unrolled Generative Adversarial Networks
- Implicit Generation and Generalization in Energy-Based Models
- Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
- Measuring Sample Quality with Kernels
- Sliced Score Matching: A Scalable Approach to Density and Score Estimation
- Deep Energy Estimator Networks
- Kernel Exponential Family Estimation via Doubly Dual Embedding
- Stein Neural Sampler
- Sliced Kernelized Stein Discrepancy
- Learning Implicit Generative Models by Teaching Explicit Ones
- Variational Noise-Contrastive Estimation