Multi-label Contrastive Predictive Coding

Abstract

Variational mutual information (MI) estimators are widely used in unsupervised representation learning methods such as contrastive predictive coding (CPC). A lower bound on MI can be obtained from a multi-class classification problem, where a critic attempts to distinguish a positive sample drawn from the underlying joint distribution from $(m-1)$ negative samples drawn from a suitable proposal distribution. Using this approach, MI estimates are bounded above by $\log m$, and could thus severely underestimate unless $m$ is very large. To overcome this limitation, we introduce a novel estimator based on a multi-label classification problem, where the critic needs to jointly identify multiple positive samples at the same time. We show that using the same amount of negative samples, multi-label CPC is able to exceed the $\log m$ bound, while still being a valid lower bound of mutual information. We demonstrate that the proposed approach is able to lead to better mutual information estimation, gain empirical improvements in unsupervised representation learning, and beat a current state-of-the-art knowledge distillation method over 10 out of 13 tasks.

arXiv commentPost camera-ready version. Reorganized the theorems in the last version as corollaries of more general theorems

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Language Models are Few-Shot Learners
• Improved Baselines with Momentum Contrastive Learning
• Conditional Image Generation with PixelCNN Decoders
• Learning deep representations by mutual information estimation and maximization
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Contrastive Multiview Coding
• MINE: Mutual Information Neural Estimation
• Large-Margin Softmax Loss for Convolutional Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
• Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
• Rethinking Feature Discrimination and Polymerization for Large-scale Recognition
• Stronger Data Poisoning Attacks Break Data Sanitization Defenses
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Learning Controllable Fair Representations
• Understanding the Limitations of Variational Mutual Information Estimators
• Striking the Right Balance with Uncertainty
• Overfitting of neural nets under class imbalance: Analysis and improvements for segmentation
• Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency

Cited by in corpus (5)

• Multi-Label Contrastive Learning for Abstract Visual Reasoning
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Integrating Auxiliary Information in Self-supervised Learning
• Contrastive Learning of Global-Local Video Representations
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation