Bounds all around: training energy-based models with bidirectional bounds
arXiv:2111.00929
Abstract
Energy-based models (EBMs) provide an elegant framework for density estimation, but they are notoriously difficult to train. Recent work has established links to generative adversarial networks, where the EBM is trained through a minimax game with a variational value function. We propose a bidirectional bound on the EBM log-likelihood, such that we maximize a lower bound and minimize an upper bound when solving the minimax game. We link one bound to a gradient penalty that stabilizes training, thereby providing grounding for best engineering practice. To evaluate the bounds we develop a new and efficient estimator of the Jacobi-determinant of the EBM generator. We demonstrate that these developments significantly stabilize training and yield high-quality density estimation and sample generation.
arXiv commentThis paper has been accepted by NeurIPS 2021
References in corpus (14)
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- Deep Anomaly Detection with Outlier Exposure
- Assessing Generative Models via Precision and Recall
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
- ContraGAN: Contrastive Learning for Conditional Image Generation
- Improving Generalization and Stability of Generative Adversarial Networks
- Deep Directed Generative Models with Energy-Based Probability Estimation
- Maximum Entropy Generators for Energy-Based Models
- Uncertainty in the Variational Information Bottleneck
- Calibrating Energy-based Generative Adversarial Networks
- The Geometry of Deep Generative Image Models and its Applications
- Generative Adversarial Networks as Variational Training of Energy Based Models
- Boltzmann machines and energy-based models
- A deep learning theory for neural networks grounded in physics