Improved Contrastive Divergence Training of Energy Based Models

Abstract

Contrastive divergence is a popular method of training energy-based models, but is known to have difficulties with training stability. We propose an adaptation to improve contrastive divergence training by scrutinizing a gradient term that is difficult to calculate and is often left out for convenience. We show that this gradient term is numerically significant and in practice is important to avoid training instabilities, while being tractable to estimate. We further highlight how data augmentation and multi-scale processing can be used to improve model robustness and generation quality. Finally, we empirically evaluate stability of model architectures and show improved performance on a host of benchmarks and use cases,such as image generation, OOD detection, and compositional generation.

arXiv commentICML 2021, Project webpage at https://energy-based-model.github.io/improved-contrastive-divergence

References in corpus (21)

• A Simple Framework for Contrastive Learning of Visual Representations
• Denoising Diffusion Probabilistic Models
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Pixel Recurrent Neural Networks
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Generative Modeling by Estimating Gradients of the Data Distribution
• Reinforcement Learning with Deep Energy-Based Policies
• A Theory of Generative ConvNet
• Implicit Generation and Generalization in Energy-Based Models
• Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Deep Energy Estimator Networks
• Autoregressive Quantile Networks for Generative Modeling
• Meta-Learning Deep Energy-Based Memory Models
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE
• Training Deep Energy-Based Models with f-Divergence Minimization
• Energy-Based Models for Continual Learning

Cited by in corpus (6)

• Pre-training Molecular Graph Representation with 3D Geometry
• How to Train Your Energy-Based Models
• Designing Perceptual Puzzles by Differentiating Probabilistic Programs
• Compositional Visual Generation and Inference with Energy Based Models
• Graph Energy-based Model for Substructure Preserving Molecular Design
• Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling