How to Train Your Energy-Based Models

Abstract

Energy-Based Models (EBMs), also known as non-normalized probabilistic models, specify probability density or mass functions up to an unknown normalizing constant. Unlike most other probabilistic models, EBMs do not place a restriction on the tractability of the normalizing constant, thus are more flexible to parameterize and can model a more expressive family of probability distributions. However, the unknown normalizing constant of EBMs makes training particularly difficult. Our goal is to provide a friendly introduction to modern approaches for EBM training. We start by explaining maximum likelihood training with Markov chain Monte Carlo (MCMC), and proceed to elaborate on MCMC-free approaches, including Score Matching (SM) and Noise Constrastive Estimation (NCE). We highlight theoretical connections among these three approaches, and end with a brief survey on alternative training methods, which are still under active research. Our tutorial is targeted at an audience with basic understanding of generative models who want to apply EBMs or start a research project in this direction.

References in corpus (23)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Denoising Diffusion Probabilistic Models
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Variational Inference with Normalizing Flows
• Density estimation using Real NVP
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Reinforcement Learning with Deep Energy-Based Policies
• MADE: Masked Autoencoder for Distribution Estimation
• A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
• A Theory of Generative ConvNet
• Improved Techniques for Training Score-Based Generative Models
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Measuring Sample Quality with Stein's Method
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Maximum Entropy Generators for Energy-Based Models
• Unbiased Markov chain Monte Carlo with couplings
• Improved Contrastive Divergence Training of Energy Based Models
• Kernel Exponential Family Estimation via Doubly Dual Embedding
• Generative Adversarial Networks as Variational Training of Energy Based Models
• Residual Energy-Based Models for Text Generation
• Training Deep Energy-Based Models with f-Divergence Minimization

Cited by in corpus (17)

• Deep Learning for Time Series Forecasting: Tutorial and Literature Survey
• Pre-training Molecular Graph Representation with 3D Geometry
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Learning to Efficiently Sample from Diffusion Probabilistic Models
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Learning Implicit Priors for Motion Optimization
• ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models
• EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models
• On tuning consistent annealed sampling for denoising score matching
• Scaling up Continuous-Time Markov Chains Helps Resolve Underspecification
• Implicit Behavioral Cloning
• Reconstruction of Pairwise Interactions using Energy-Based Models
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
• Autoencoding Under Normalization Constraints
• Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling
• Learning Proposals for Practical Energy-Based Regression