Momentum Contrast for Unsupervised Visual Representation Learning

Abstract

We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a moving-averaged encoder. This enables building a large and consistent dictionary on-the-fly that facilitates contrastive unsupervised learning. MoCo provides competitive results under the common linear protocol on ImageNet classification. More importantly, the representations learned by MoCo transfer well to downstream tasks. MoCo can outperform its supervised pre-training counterpart in 7 detection/segmentation tasks on PASCAL VOC, COCO, and other datasets, sometimes surpassing it by large margins. This suggests that the gap between unsupervised and supervised representation learning has been largely closed in many vision tasks.

CVPR 2020 camera-ready. Code: https://github.com/facebookresearch/moco

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Improved Baselines with Momentum Contrastive Learning
• Learning deep representations by mutual information estimation and maximization
• YFCC100M: The New Data in Multimedia Research
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• Large Scale Adversarial Representation Learning
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Unsupervised Embedding Learning via Invariant and Spreading Instance Feature

Cited by in corpus (182)

• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• COVID-CT-Dataset: A CT Scan Dataset about COVID-19
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Exploring Simple Siamese Representation Learning
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• An Overview of Deep Semi-Supervised Learning
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Universal Domain Adaptation through Self Supervision
• Audiovisual SlowFast Networks for Video Recognition
• Self-Supervised Learning for Videos: A Survey
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Contrastive Code Representation Learning
• Image Augmentations for GAN Training
• A Comprehensive Study of Deep Video Action Recognition
• Supervised Contrastive Learning
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Domain Generalization using Causal Matching
• Subject-Aware Contrastive Learning for Biosignals
• Robust Pre-Training by Adversarial Contrastive Learning
• GPT-GNN: Generative Pre-Training of Graph Neural Networks
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Intriguing Properties of Contrastive Losses
• Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Contrastive Learning with Adversarial Examples
• MixCo: Mix-up Contrastive Learning for Visual Representation
• MoCo-CXR: MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Contrastive Training for Improved Out-of-Distribution Detection
• Self-supervised Action Representation Learning from Partial Spatio-Temporal Skeleton Sequences
• Unsupervised Deep Representation Learning and Few-Shot Classification of PolSAR Images
• Object DGCNN: 3D Object Detection using Dynamic Graphs
• On Mutual Information in Contrastive Learning for Visual Representations
• Self-Supervision Closes the Gap Between Weak and Strong Supervision in Histology
• Knowledge-Aware Language Model Pretraining
• Towards an astronomical foundation model for stars with a Transformer-based model
• Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval
• Generative Image Inpainting with Segmentation Confusion Adversarial Training and Contrastive Learning
• Whitening for Self-Supervised Representation Learning
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Representation Learning via Invariant Causal Mechanisms
• Propagation Tree Is Not Deep: Adaptive Graph Contrastive Learning Approach for Rumor Detection
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Improving Semantic Segmentation via Self-Training
• MoPro: Webly Supervised Learning with Momentum Prototypes
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Contrastive Learning of Musical Representations
• Cross-domain Self-supervised Learning for Domain Adaptation with Few Source Labels
• Uncertainty Quantification and Deep Ensembles
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
• Semi-Supervised Contrastive Learning with Generalized Contrastive Loss and Its Application to Speaker Recognition
• MST: Masked Self-Supervised Transformer for Visual Representation
• Self-Supervised Relational Reasoning for Representation Learning
• Improving out-of-distribution generalization via multi-task self-supervised pretraining
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• Understanding the Behaviour of Contrastive Loss
• Deep Reinforcement and InfoMax Learning
• A Review of Single-Source Deep Unsupervised Visual Domain Adaptation
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Learning Representations from Audio-Visual Spatial Alignment
• Self-Supervised Learning with Kernel Dependence Maximization
• Joint Contrastive Learning for Unsupervised Domain Adaptation
• Unsupervised Representation Learning by InvariancePropagation
• The Dawn of AI-Native EDA: Opportunities and Challenges of Large Circuit Models
• LoCo: Local Contrastive Representation Learning
• Deep Nets: What have they ever done for Vision?
• Contrastive Visual-Linguistic Pretraining
• Diversity Helps: Unsupervised Few-shot Learning via Distribution Shift-based Data Augmentation
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Motif-Driven Contrastive Learning of Graph Representations
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• CAPT: Contrastive Pre-Training for Learning Denoised Sequence Representations
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Image Deconvolution via Noise-Tolerant Self-Supervised Inversion
• Proposal Learning for Semi-Supervised Object Detection
• Distilling Localization for Self-Supervised Representation Learning
• Contrastive Learning for Debiased Candidate Generation in Large-Scale Recommender Systems
• Comparing to Learn: Surpassing ImageNet Pretraining on Radiographs By Comparing Image Representations
• Implicit Rank-Minimizing Autoencoder
• On Learning Universal Representations Across Languages
• Self-supervised Learning from a Multi-view Perspective
• Being Bayesian about Categorical Probability
• MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering
• Self-supervised graph neural networks for accurate prediction of Néel temperature
• FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning
• Improving Transformation Invariance in Contrastive Representation Learning
• Multi-label Contrastive Predictive Coding
• Evaluating representations by the complexity of learning low-loss predictors
• Negative Data Augmentation
• Subclass Distillation
• Supermasks in Superposition
• Learning Speaker Embedding with Momentum Contrast
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Camera-aware Proxies for Unsupervised Person Re-Identification
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data
• Agent-Centric Representations for Multi-Agent Reinforcement Learning
• Online Deep Clustering for Unsupervised Representation Learning
• RetCL: A Selection-based Approach for Retrosynthesis via Contrastive Learning
• A Transductive Approach for Video Object Segmentation
• Artificial Immune System of Secure Face Recognition Against Adversarial Attacks
• Neural Methods for Point-wise Dependency Estimation
• Meta Approach to Data Augmentation Optimization
• CoKe: Localized Contrastive Learning for Robust Keypoint Detection
• The Phonexia VoxCeleb Speaker Recognition Challenge 2021 System Description
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
• Action-Based Representation Learning for Autonomous Driving
• MomentumRNN: Integrating Momentum into Recurrent Neural Networks
• Unsupervised domain adaptation via coarse-to-fine feature alignment method using contrastive learning
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery
• Enhancing Dialogue Generation via Multi-Level Contrastive Learning
• Cheaper Pre-training Lunch: An Efficient Paradigm for Object Detection
• Function Contrastive Learning of Transferable Meta-Representations
• Cross-Domain Sentiment Classification with In-Domain Contrastive Learning
• AutoLTS: Automating Cycling Stress Assessment via Contrastive Learning and Spatial Post-processing
• Do Explicit Alignments Robustly Improve Multilingual Encoders?
• Self-supervised Visual Attribute Learning for Fashion Compatibility
• Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Triplet Contrastive Learning for Brain Tumor Classification
• FedSC: Federated Learning with Semantic-Aware Collaboration
• Multi-label Iterated Learning for Image Classification with Label Ambiguity
• General-Purpose User Embeddings based on Mobile App Usage
• Jigsaw Clustering for Unsupervised Visual Representation Learning
• Mapping in a cycle: Sinkhorn regularized unsupervised learning for point cloud shapes
• StackMix: A complementary Mix algorithm
• Levenshtein Distance Embedding with Poisson Regression for DNA Storage
• Long-range and hierarchical language predictions in brains and algorithms
• Cross-domain Transfer of defect features in technical domains based on partial target data
• Unsupervised Image Classification for Deep Representation Learning
• Self-supervised Feature Learning via Exploiting Multi-modal Data for Retinal Disease Diagnosis
• Multi-view Contrastive Learning for Online Knowledge Distillation
• Self-Supervised Learning from Unlabeled Fundus Photographs Improves Segmentation of the Retina
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Disentangle Perceptual Learning through Online Contrastive Learning
• Improving Object Detection with Selective Self-supervised Self-training
• 3D Object Recognition By Corresponding and Quantizing Neural 3D Scene Representations
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations
• Self-supervised Learning for Semi-supervised Temporal Language Grounding
• Unsupervised Person Re-Identification with Multi-Label Learning Guided Self-Paced Clustering
• LabelEnc: A New Intermediate Supervision Method for Object Detection
• Feature Distillation With Guided Adversarial Contrastive Learning
• Self-supervised Regularization for Text Classification
• Jointly Learnable Data Augmentations for Self-Supervised GNNs
• Self-Supervised Dynamic Networks for Covariate Shift Robustness
• Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations
• Towards Robust Pattern Recognition: A Review
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Cross-Domain Sentiment Classification with Contrastive Learning and Mutual Information Maximization
• Using Python for Model Inference in Deep Learning
• Multi-Pretext Attention Network for Few-shot Learning with Self-supervision
• QK Iteration: A Self-Supervised Representation Learning Algorithm for Image Similarity
• Cross-modal Image Retrieval with Deep Mutual Information Maximization
• Atlas Based Representation and Metric Learning on Manifolds
• Voxel-level Siamese Representation Learning for Abdominal Multi-Organ Segmentation
• Longitudinal Self-Supervised Learning
• SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval
• NeMo: A Neuron-Level Modularizing-While-Training Approach for Decomposing DNN Models
• Is High Variance Unavoidable in RL? A Case Study in Continuous Control
• Data-Driven Self-Supervised Graph Representation Learning
• Synthesizing Photorealistic Images with Deep Generative Learning
• SVMax: A Feature Embedding Regularizer
• Learning Rich Nearest Neighbor Representations from Self-supervised Ensembles
• LiDAM: Semi-Supervised Learning with Localized Domain Adaptation and Iterative Matching
• A Review of Computer Vision Methods in Network Security
• Network representation learning: A macro and micro view