Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling

Abstract

We present a new method for evaluating and training unnormalized density models. Our approach only requires access to the gradient of the unnormalized model's log-density. We estimate the Stein discrepancy between the data density $p(x)$ and the model density $q(x)$ defined by a vector function of the data. We parameterize this function with a neural network and fit its parameters to maximize the discrepancy. This yields a novel goodness-of-fit test which outperforms existing methods on high dimensional data. Furthermore, optimizing $q(x)$ to minimize this discrepancy produces a novel method for training unnormalized models which scales more gracefully than existing methods. The ability to both learn and compare models is a unique feature of the proposed method.

ICML 2020

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Generative Modeling by Estimating Gradients of the Data Distribution
• Searching for Activation Functions
• A note on the evaluation of generative models
• A Theory of Generative ConvNet
• Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Implicit Generation and Generalization in Energy-Based Models
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Measuring Sample Quality with Kernels
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Explorations in Homeomorphic Variational Auto-Encoding
• Operator Variational Inference
• A Linear-Time Kernel Goodness-of-Fit Test
• Deep Energy Estimator Networks
• Unbiased Markov chain Monte Carlo with couplings
• The Shape of Data: Intrinsic Distance for Data Distributions
• Stein Neural Sampler
• Flow Contrastive Estimation of Energy-Based Models

Cited by in corpus (14)

• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Sliced Kernelized Stein Discrepancy
• Oops I Took A Gradient: Scalable Sampling for Discrete Distributions
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Telescoping Density-Ratio Estimation
• On Energy-Based Models with Overparametrized Shallow Neural Networks
• Stein Variational Gaussian Processes
• A Fourier representation of kernel Stein discrepancy with application to Goodness-of-Fit tests for measures on infinite dimensional Hilbert spaces
• Composite Goodness-of-fit Tests with Kernels
• Active Slices for Sliced Stein Discrepancy
• Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More
• Generative Particle Variational Inference via Estimation of Functional Gradients
• Learning Deep Representation with Energy-Based Self-Expressiveness for Subspace Clustering
• Particle Dynamics for Learning EBMs