Energy-Inspired Models: Learning with Sampler-Induced Distributions
arXiv:1910.14265
Abstract
Energy-based models (EBMs) are powerful probabilistic models, but suffer from intractable sampling and density evaluation due to the partition function. As a result, inference in EBMs relies on approximate sampling algorithms, leading to a mismatch between the model and inference. Motivated by this, we consider the sampler-induced distribution as the model of interest and maximize the likelihood of this model. This yields a class of energy-inspired models (EIMs) that incorporate learned energy functions while still providing exact samples and tractable log-likelihood lower bounds. We describe and evaluate three instantiations of such models based on truncated rejection sampling, self-normalized importance sampling, and Hamiltonian importance sampling. These models outperform or perform comparably to the recently proposed Learned Accept/Reject Sampling algorithm and provide new insights on ranking Noise Contrastive Estimation and Contrastive Predictive Coding. Moreover, EIMs allow us to generalize a recent connection between multi-sample variational lower bounds and auxiliary variable variational inference. We show how recent variational bounds can be unified with EIMs as the variational family.
arXiv commentPresented at the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada
References in corpus (26)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Categorical Reparameterization with Gumbel-Softmax
- Variational Inference with Normalizing Flows
- Exploring the Limits of Language Modeling
- Density estimation using Real NVP
- A Recurrent Latent Variable Model for Sequential Data
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Stochastic Video Generation with a Learned Prior
- Auxiliary Deep Generative Models
- Variational Lossy Autoencoder
- A Theory of Generative ConvNet
- Deep Kalman Filters
- Implicit Generation and Generalization in Energy-Based Models
- Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
- PixelVAE: A Latent Variable Model for Natural Images
- Deep Directed Generative Models with Energy-Based Probability Estimation
- Auto-Encoding Sequential Monte Carlo
- Reinterpreting Importance-Weighted Autoencoders
- Importance Weighting and Variational Inference
- Semi-Implicit Variational Inference
- Kernel Exponential Family Estimation via Doubly Dual Embedding
- Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
- Resampled Priors for Variational Autoencoders
- Doubly Semi-Implicit Variational Inference