Training Deep Energy-Based Models with f-Divergence Minimization

Abstract

Deep energy-based models (EBMs) are very flexible in distribution parametrization but computationally challenging because of the intractable partition function. They are typically trained via maximum likelihood, using contrastive divergence to approximate the gradient of the KL divergence between data and model distribution. While KL divergence has many desirable properties, other f-divergences have shown advantages in training implicit density generative models such as generative adversarial networks. In this paper, we propose a general variational framework termed f-EBM to train EBMs using any desired f-divergence. We introduce a corresponding optimization algorithm and prove its local convergence property with non-linear dynamical systems theory. Experimental results demonstrate the superiority of f-EBM over contrastive divergence, as well as the benefits of training EBMs using f-divergences other than KL.

arXiv commentICML 2020

References in corpus (21)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• WaveNet: A Generative Model for Raw Audio
• Spectral Normalization for Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Which Training Methods for GANs do actually Converge?
• Density estimation using Real NVP
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• MADE: Masked Autoencoder for Distribution Estimation
• Learning in Implicit Generative Models
• Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
• Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
• Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
• Maximum Entropy Generators for Energy-Based Models
• Kernel Exponential Family Estimation via Doubly Dual Embedding
• Exponential Family Estimation via Adversarial Dynamics Embedding
• To Relieve Your Headache of Training an MRF, Take AdVIL
• Noise contrastive estimation: asymptotics, comparison with MC-MLE

Cited by in corpus (9)

• How to Train Your Energy-Based Models
• Improved Contrastive Divergence Training of Energy Based Models
• $f$-GAIL: Learning $f$-Divergence for Generative Adversarial Imitation Learning
• Telescoping Density-Ratio Estimation
• Generalized Energy Based Models
• Autoregressive Score Matching
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Imitation with Neural Density Models
• Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation