Hybrid Discriminative-Generative Training via Contrastive Learning

Abstract

Contrastive learning and supervised learning have both seen significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification accuracy of WideResNet on CIFAR-10 and CIFAR-100. It also leads to improved performance on robustness, out-of-distribution detection, and calibration.

Code: https://github.com/lhao499/HDGE

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distributed Representations of Words and Phrases and their Compositionality
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Bootstrap your own latent: A new approach to self-supervised Learning
• Spectral Normalization for Generative Adversarial Networks
• Improved Baselines with Momentum Contrastive Learning
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Momentum Contrast for Unsupervised Visual Representation Learning
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Residual Flows for Invertible Generative Modeling
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• A Theory of Generative ConvNet
• Supervised Contrastive Learning
• Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Contrastive Representation Distillation
• Introspective Classification with Convolutional Nets

Cited by in corpus (10)

• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Hard Negative Mixing for Contrastive Learning
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Perfect density models cannot guarantee anomaly detection
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models
• Do We Really Need to Learn Representations from In-domain Data for Outlier Detection?
• Hybrid Generative-Contrastive Representation Learning
• Types of Out-of-Distribution Texts and How to Detect Them