Temperature as Uncertainty in Contrastive Learning
arXiv:2110.04403
Abstract
Contrastive learning has demonstrated great capability to learn representations without annotations, even outperforming supervised baselines. However, it still lacks important properties useful for real-world application, one of which is uncertainty. In this paper, we propose a simple way to generate uncertainty scores for many contrastive methods by re-purposing temperature, a mysterious hyperparameter used for scaling. By observing that temperature controls how sensitive the objective is to specific embedding locations, we aim to learn temperature as an input-dependent variable, treating it as a measure of embedding confidence. We call this approach "Temperature as Uncertainty", or TaU. Through experiments, we demonstrate that TaU is useful for out-of-distribution detection, while remaining competitive with benchmarks on linear evaluation. Moreover, we show that TaU can be learned on top of pretrained models, enabling uncertainty scores to be generated post-hoc with popular off-the-shelf models. In summary, TaU is a simple yet versatile method for generating uncertainties for contrastive learning. Open source code can be found at: https://github.com/mhw32/temperature-as-uncertainty-public.
4 pages content; 1 page supplement
References in corpus (11)
- Learning Transferable Visual Models From Natural Language Supervision
- Bootstrap your own latent: A new approach to self-supervised Learning
- Learning Representations by Maximizing Mutual Information Across Views
- Large Batch Training of Convolutional Networks
- On Variational Bounds of Mutual Information
- Local Aggregation for Unsupervised Learning of Visual Embeddings
- Self-supervised Knowledge Distillation for Few-shot Learning
- On Mutual Information in Contrastive Learning for Visual Representations
- Conditional Negative Sampling for Contrastive Learning of Visual Representations
- Understanding the Behaviour of Contrastive Loss
- Detecting Out-of-Distribution Examples with In-distribution Examples and Gram Matrices