Prototypical Contrastive Learning of Unsupervised Representations

Abstract

This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL.

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Representation Learning with Contrastive Predictive Coding
• Prototypical Networks for Few-shot Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Learning Representations by Maximizing Mutual Information Across Views
• Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering
• Contrastive Multiview Coding
• Large Scale Adversarial Representation Learning
• On Mutual Information Maximization for Representation Learning
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Rethinking ImageNet Pre-training
• AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data

Cited by in corpus (93)

• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Contrastive Representation Learning: A Framework and Review
• Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Hard Negative Mixing for Contrastive Learning
• Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images
• Twin Contrastive Learning for Online Clustering
• Self-supervised Pretraining of Visual Features in the Wild
• Learning Representation for Clustering via Prototype Scattering and Positive Sampling
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Understanding Dimensional Collapse in Contrastive Self-supervised Learning
• Efficient Self-supervised Vision Transformers for Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• SEED: Self-supervised Distillation For Visual Representation
• Intriguing Properties of Contrastive Losses
• Unsupervised Feature Learning by Autoencoder and Prototypical Contrastive Learning for Hyperspectral Classification
• Are all negatives created equal in contrastive instance discrimination?
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Contrastive Learning with Stronger Augmentations
• Support-set bottlenecks for video-text representation learning
• MoPro: Webly Supervised Learning with Momentum Prototypes
• Understanding Self-supervised Learning with Dual Deep Networks
• P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
• Heterogeneous Graph Contrastive Learning with Meta-path Contexts and Adaptively Weighted Negative Samples
• Hyperbolic Hierarchical Contrastive Hashing
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• False: False Negative Samples Aware Contrastive Learning for Semantic Segmentation of High-Resolution Remote Sensing Image
• Hierarchical Skeleton Meta-Prototype Contrastive Learning with Hard Skeleton Mining for Unsupervised Person Re-Identification
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• X-GOAL: Multiplex Heterogeneous Graph Prototypical Contrastive Learning
• Adaptive Soft Contrastive Learning
• Unsupervised Representation Learning by InvariancePropagation
• Continual Coarse-to-Fine Domain Adaptation in Semantic Segmentation
• Multi-Label Image Classification with Contrastive Learning
• Self-supervised Pre-training with Hard Examples Improves Visual Representations
• SoundCLR: Contrastive Learning of Representations For Improved Environmental Sound Classification
• Clinically Labeled Contrastive Learning for OCT Biomarker Classification
• Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
• Joint Noise-Tolerant Learning and Meta Camera Shift Adaptation for Unsupervised Person Re-Identification
• Class Gradient Projection For Continual Learning
• Contrastive Predictive Coding for Anomaly Detection
• MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Weakly Supervised Semantic Segmentation by Pixel-to-Prototype Contrast
• A Co-training Approach for Noisy Time Series Learning
• Proxy-based Item Representation for Attribute and Context-aware Recommendation
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Prototypical Contrast and Reverse Prediction: Unsupervised Skeleton Based Action Recognition
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
• Delving Deep into the Generalization of Vision Transformers under Distribution Shifts
• Interpretable agent communication from scratch (with a generic visual processor emerging on the side)
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
• Supporting Clustering with Contrastive Learning
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Unsupervised Object-Level Representation Learning from Scene Images
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• Representation Learning via Consistent Assignment of Views to Clusters
• Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
• Prototype Memory for Large-scale Face Representation Learning
• Self-Supervised Ranking for Representation Learning
• Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
• On Feature Decorrelation in Self-Supervised Learning
• Deep Contrastive Graph Representation via Adaptive Homotopy Learning
• Center-wise Local Image Mixture For Contrastive Representation Learning
• Self-Supervised Place Recognition by Refining Temporal and Featural Pseudo Labels from Panoramic Data
• APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Jigsaw Clustering for Unsupervised Visual Representation Learning
• Experiments on Anomaly Detection in Autonomous Driving by Forward-Backward Style Transfers
• Integrating Auxiliary Information in Self-supervised Learning
• Prototypical Contrastive Learning and Adaptive Interest Selection for Candidate Generation in Recommendations
• Contrastive Learning for Self-Supervised Pre-Training of Point Cloud Segmentation Networks With Image Data
• Multi-source Few-shot Domain Adaptation
• Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning
• Don't miss the Mismatch: Investigating the Objective Function Mismatch for Unsupervised Representation Learning
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Look, Cast and Mold: Learning 3D Shape Manifold from Single-view Synthetic Data
• You Never Cluster Alone
• Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning
• Unsupervised Embedding Learning from Uncertainty Momentum Modeling
• Progressive Stage-wise Learning for Unsupervised Feature Representation Enhancement
• CROCS: Clustering and Retrieval of Cardiac Signals Based on Patient Disease Class, Sex, and Age
• Self-Supervised Pretraining and Controlled Augmentation Improve Rare Wildlife Recognition in UAV Images
• UCSL : A Machine Learning Expectation-Maximization framework for Unsupervised Clustering driven by Supervised Learning
• FROST: Faster and more Robust One-shot Semi-supervised Training
• A Practice of Post-Training on Llama-3 70B with Optimal Selection of Additional Language Mixture Ratio
• Unified Framework for Feature Extraction based on Contrastive Learning
• Online Unsupervised Learning of Visual Representations and Categories
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets