Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One

Abstract

We propose to reinterpret a standard discriminative classifier of p(y|x) as an energy based model for the joint distribution p(x,y). In this setting, the standard class probabilities can be easily computed as well as unnormalized values of p(x) and p(x|y). Within this framework, standard discriminative architectures may beused and the model can also be trained on unlabeled data. We demonstrate that energy based training of the joint distribution improves calibration, robustness, andout-of-distribution detection while also enabling our models to generate samplesrivaling the quality of recent GAN approaches. We improve upon recently proposed techniques for scaling up the training of energy based models and presentan approach which adds little overhead compared to standard classification training. Our approach is the first to achieve performance rivaling the state-of-the-artin both generative and discriminative learning within one hybrid model.

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples
• Generative Modeling by Estimating Gradients of the Data Distribution
• Synthesizing Robust Adversarial Examples
• Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks
• PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
• Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
• A Note on the Inception Score
• A Theory of Generative ConvNet
• Invertible Residual Networks
• Implicit Generation and Generalization in Energy-Based Models
• Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
• Adversarial Distillation of Bayesian Neural Network Posteriors
• Are Generative Classifiers More Robust to Adversarial Attacks?

Cited by in corpus (38)

• Diffusion Models Beat GANs on Image Synthesis
• Explaining Neural Scaling Laws
• Controversial stimuli: pitting neural networks against each other as models of human recognition
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Improved Contrastive Divergence Training of Energy Based Models
• Perfect density models cannot guarantee anomaly detection
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Unsupervised Learning of Compositional Energy Concepts
• Further Analysis of Outlier Detection with Deep Generative Models
• Generative Modeling with Denoising Auto-Encoders and Langevin Sampling
• Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
• Oops I Took A Gradient: Scalable Sampling for Discrete Distributions
• Supermasks in Superposition
• Compositional Visual Generation and Inference with Energy Based Models
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models
• Protecting Classifiers From Attacks
• Elsa: Energy-based learning for semi-supervised anomaly detection
• Training Normalizing Flows with the Information Bottleneck for Competitive Generative Classification
• Energy-Based Models for Continual Learning
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• Generalized Energy Based Models
• Overcoming Catastrophic Forgetting by Generative Regularization
• Learning from the Best: Rationalizing Prediction by Adversarial Information Calibration
• Exponential Tilting of Generative Models: Improving Sample Quality by Training and Sampling from Latent Energy
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning
• On the human-recognizability phenomenon of adversarially trained deep image classifiers
• Energy-Based Anomaly Detection and Localization
• Likelihood Landscapes: A Unifying Principle Behind Many Adversarial Defenses
• Analyzing and Improving the Optimization Landscape of Noise-Contrastive Estimation
• Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling
• RNA Alternative Splicing Prediction with Discrete Compositional Energy Network
• Particle Dynamics for Learning EBMs
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty
• An Energy-Based View of Graph Neural Networks