Learning Proposals for Practical Energy-Based Regression

Abstract

Energy-based models (EBMs) have experienced a resurgence within machine learning in recent years, including as a promising alternative for probabilistic regression. However, energy-based regression requires a proposal distribution to be manually designed for training, and an initial estimate has to be provided at test-time. We address both of these issues by introducing a conceptually simple method to automatically learn an effective proposal distribution, which is parameterized by a separate network head. To this end, we derive a surprising result, leading to a unified training objective that jointly minimizes the KL divergence from the proposal to the EBM, and the negative log-likelihood of the EBM. At test-time, we can then employ importance sampling with the trained proposal to efficiently evaluate the learned EBM and produce stand-alone predictions. Furthermore, we utilize our derived training objective to learn mixture density networks (MDNs) with a jointly trained energy-based teacher, consistently outperforming conventional MDN training on four real-world regression tasks within computer vision. Code is available at https://github.com/fregu856/ebms_proposals.

arXiv commentAISTATS 2022. Code is available at https://github.com/fregu856/ebms_proposals

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• Exploring the Limits of Language Modeling
• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction
• A Theory of Generative ConvNet
• Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
• How to Train Your Energy-Based Models
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Maximum Entropy Generators for Energy-Based Models
• Generative Adversarial Networks as Variational Training of Energy Based Models
• Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold
• Modelling heterogeneous distributions with an Uncountable Mixture of Asymmetric Laplacians
• Learning Energy-Based Model with Variational Auto-Encoder as Amortized Sampler