A Simple Framework for Uncertainty in Contrastive Learning
arXiv:2010.02038
Abstract
Contrastive approaches to representation learning have recently shown great promise. In contrast to generative approaches, these contrastive models learn a deterministic encoder with no notion of uncertainty or confidence. In this paper, we introduce a simple approach based on "contrasting distributions" that learns to assign uncertainty for pretrained contrastive representations. In particular, we train a deep network from a representation to a distribution in representation space, whose variance can be used as a measure of confidence. In our experiments, we show that this deep uncertainty model can be used (1) to visually interpret model behavior, (2) to detect new noise in the input to deployed models, (3) to detect anomalies, where we outperform 10 baseline methods across 11 tasks with improvements of up to 14% absolute, and (4) to classify out-of-distribution examples where our fully unsupervised model is competitive with supervised methods.
8 pages main text
References in corpus (23)
- Scikit-learn: Machine Learning in Python
- Auto-Encoding Variational Bayes
- A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap your own latent: A new approach to self-supervised Learning
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Improved Baselines with Momentum Contrastive Learning
- A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Deep Learning for Anomaly Detection: A Survey
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
- Adversarially Learned Inference
- Contrastive Multiview Coding
- Deep Anomaly Detection with Outlier Exposure
- InfoVAE: Information Maximizing Variational Autoencoders
- Grad-CAM: Why did you say that?
- Learning Representations of Ultrahigh-dimensional Data for Random Distance-based Outlier Detection
- Towards a Neural Statistician
- Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions
- Attention Interpretability Across NLP Tasks
- On Mutual Information in Contrastive Learning for Visual Representations
- Generative Models of Visually Grounded Imagination
- The Variational Homoencoder: Learning to learn high capacity generative models from few examples
- Meta-Amortized Variational Inference and Learning