Implicit Generation and Generalization in Energy-Based Models

Abstract

Energy based models (EBMs) are appealing due to their generality and simplicity in likelihood modeling, but have been traditionally difficult to train. We present techniques to scale MCMC based EBM training on continuous neural networks, and we show its success on the high-dimensional data domains of ImageNet32x32, ImageNet128x128, CIFAR-10, and robotic hand trajectories, achieving better samples than other likelihood models and nearing the performance of contemporary GAN approaches, while covering all modes of the data. We highlight some unique capabilities of implicit generation such as compositionality and corrupt image reconstruction and inpainting. Finally, we show that EBMs are useful models across a wide variety of tasks, achieving state-of-the-art out-of-distribution classification, adversarially robust classification, state-of-the-art continual online class learning, and coherent long term predicted trajectory rollouts.

References in corpus (25)

• Auto-Encoding Variational Bayes
• Overcoming catastrophic forgetting in neural networks
• Playing Atari with Deep Reinforcement Learning
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Pixel Recurrent Neural Networks
• Towards Deep Learning Models Resistant to Adversarial Attacks
• NICE: Non-linear Independent Components Estimation
• Continual Learning Through Synaptic Intelligence
• Energy-based Generative Adversarial Network
• Deep Anomaly Detection with Outlier Exposure
• Learning Dexterous In-Hand Manipulation
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• Towards Robust Evaluations of Continual Learning
• A Theory of Generative ConvNet
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• RNADE: The real-valued neural autoregressive density-estimator
• Deep Directed Generative Models with Energy-Based Probability Estimation
• Stein Variational Policy Gradient
• Maximum Entropy Generators for Energy-Based Models
• Autoregressive Quantile Networks for Generative Modeling
• Learning Intrinsic Image Decomposition from Watching the World
• Learning FRAME Models Using CNN Filters

Cited by in corpus (78)

• Diffusion Models Beat GANs on Image Synthesis
• Generative Modeling by Estimating Gradients of the Data Distribution
• Energy-based Out-of-distribution Detection
• Improved Techniques for Training Score-Based Generative Models
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Quantum Hamiltonian-Based Models and the Variational Quantum Thermalizer Algorithm
• A Perspective on Deep Learning for Molecular Modeling and Simulations
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Real or Fake? Learning to Discriminate Machine from Human Generated Text
• Learning Neural Generative Dynamics for Molecular Conformation Generation
• Improved Contrastive Divergence Training of Energy Based Models
• Meta-Learning Deep Energy-Based Memory Models
• Residual Energy-Based Models for Text
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Residual Energy-Based Models for Text Generation
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Exponential Family Estimation via Adversarial Dynamics Embedding
• Energy-based models for atomic-resolution protein conformations
• Energy-Based Imitation Learning
• Unsupervised Learning of Compositional Energy Concepts
• Learning Gradient Fields for Shape Generation
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• Symbolic Music Generation with Diffusion Models
• Negative Data Augmentation
• Oops I Took A Gradient: Scalable Sampling for Discrete Distributions
• Orthogonal Convolutional Neural Networks
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models
• Compositional Visual Generation and Inference with Energy Based Models
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• Energy-Based Models for Continual Learning
• Convolutional Bipartite Attractor Networks
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• Generative Modeling with Optimal Transport Maps
• Trajectory Prediction with Latent Belief Energy-Based Model
• On Energy-Based Models with Overparametrized Shallow Neural Networks
• Improved Autoregressive Modeling with Distribution Smoothing
• Fast Convergence for Langevin Diffusion with Manifold Structure
• A Neural Network MCMC sampler that maximizes Proposal Entropy
• Overcoming Catastrophic Forgetting by Generative Regularization
• Learning Latent Space Energy-Based Prior Model
• Flow Contrastive Estimation of Energy-Based Models
• Model Based Planning with Energy Based Models
• Exponential Tilting of Generative Models: Improving Sample Quality by Training and Sampling from Latent Energy
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• An Algorithm for Out-Of-Distribution Attack to Neural Network Encoder
• Energy-constrained Self-training for Unsupervised Domain Adaptation
• Regularizing Model-Based Planning with Energy-Based Models
• Joint Training of Variational Auto-Encoder and Latent Energy-Based Model
• Energy-Based Processes for Exchangeable Data
• Contrastive Divergence Learning is a Time Reversal Adversarial Game
• Set Distribution Networks: a Generative Model for Sets of Images
• On Out-of-distribution Detection with Energy-based Models
• Gradient Origin Networks
• Reconstruction of Pairwise Interactions using Energy-Based Models
• Autoregressive Score Matching
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Energy-Based Anomaly Detection and Localization
• Lattice Protein Folding with Variational Annealing
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
• Ordering Dimensions with Nested Dropout Normalizing Flows
• Imitation with Neural Density Models
• RNA Alternative Splicing Prediction with Discrete Compositional Energy Network
• Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling
• Pre-Training Transformers as Energy-Based Cloze Models
• Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation
• An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
• Cascading Modular Network (CAM-Net) for Multimodal Image Synthesis
• Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More
• NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• Entropic Issues in Likelihood-Based OOD Detection
• Particle Dynamics for Learning EBMs