Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
arXiv:2107.01152
Abstract
InfoNCE-based contrastive representation learners, such as SimCLR, have been tremendously successful in recent years. However, these contrastive schemes are notoriously resource demanding, as their effectiveness breaks down with small-batch training (i.e., the log-K curse, whereas K is the batch-size). In this work, we reveal mathematically why contrastive learners fail in the small-batch-size regime, and present a novel simple, non-trivial contrastive objective named FlatNCE, which fixes this issue. Unlike InfoNCE, our FlatNCE no longer explicitly appeals to a discriminative classification goal for contrastive learning. Theoretically, we show FlatNCE is the mathematical dual formulation of InfoNCE, thus bridging the classical literature on energy modeling; and empirically, we demonstrate that, with minimal modification of code, FlatNCE enables immediate performance boost independent of the subject-matter engineering efforts. The significance of this work is furthered by the powerful generalization of contrastive learning techniques, and the introduction of new tools to monitor and diagnose contrastive training. We substantiate our claims with empirical evidence on CIFAR10, ImageNet, and other datasets, where FlatNCE consistently outperforms InfoNCE.
References in corpus (21)
- Distributed Representations of Words and Phrases and their Compositionality
- Learning Transferable Visual Models From Natural Language Supervision
- Estimating Mutual Information
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Learning deep representations by mutual information estimation and maximization
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Contrastive Representation Learning: A Framework and Review
- Estimating divergence functionals and the likelihood ratio by convex risk minimization
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- Contrastive Multiview Coding
- An efficient framework for learning sentence representations
- Hard Negative Mixing for Contrastive Learning
- A Theoretical Analysis of Contrastive Unsupervised Representation Learning
- On Variational Bounds of Mutual Information
- Formal Limitations on the Measurement of Mutual Information
- CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information
- Contrastive Learning with Hard Negative Samples
- Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning
- Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
- Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency