Contrastive Multiview Coding

Abstract

Humans view the world through many sensory channels, e.g., the long-wavelength light channel, viewed by the left eye, or the high-frequency vibrations channel, heard by the right ear. Each view is noisy and incomplete, but important factors, such as physics, geometry, and semantics, tend to be shared between all views (e.g., a "dog" can be seen, heard, and felt). We investigate the classic hypothesis that a powerful representation is one that models view-invariant factors. We study this hypothesis under the framework of multiview contrastive learning, where we learn a representation that aims to maximize mutual information between different views of the same scene but is otherwise compact. Our approach scales to any number of views, and is view-agnostic. We analyze key properties of the approach that make it work, finding that the contrastive loss outperforms a popular alternative based on cross-view prediction, and that the more views we learn from, the better the resulting representation captures underlying scene semantics. Our approach achieves state-of-the-art results on image and video unsupervised learning benchmarks. Code is released at: http://github.com/HobbitLong/CMC/.

arXiv commentCode: http://github.com/HobbitLong/CMC/

References in corpus (24)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Representation Learning with Contrastive Predictive Coding
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Learning deep representations by mutual information estimation and maximization
• Pixel Recurrent Neural Networks
• Momentum Contrast for Unsupervised Visual Representation Learning
• A Survey on Multi-view Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Learning Representations by Maximizing Mutual Information Across Views
• MINE: Mutual Information Neural Estimation
• Large Scale Adversarial Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• Unsupervised Learning by Predicting Noise
• Data-dependent Initializations of Convolutional Neural Networks
• On Variational Bounds of Mutual Information
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Learning visual groups from co-occurrences in space and time
• Formal Limitations on the Measurement of Mutual Information
• Similarity Learning for Provably Accurate Sparse Linear Classification
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
• Learning Robust and Multilingual Speech Representations
• Unsupervised Learning from Video with Deep Neural Embeddings
• Evolving Losses for Unlabeled Video Representation Learning

Cited by in corpus (320)

• A Simple Framework for Contrastive Learning of Visual Representations
• Learning Transferable Visual Models From Natural Language Supervision
• Transformers in Vision: A Survey
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Contrastive Representation Learning: A Framework and Review
• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• Prototypical Contrastive Learning of Unsupervised Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• A Review on Deep Learning in UAV Remote Sensing
• Exploring Simple Siamese Representation Learning
• Deep Graph Contrastive Representation Learning
• Rethinking Pre-training and Self-training
• Contrastive Multi-View Representation Learning on Graphs
• XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• On Mutual Information Maximization for Representation Learning
• Hard Negative Mixing for Contrastive Learning
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• CLEAR: Contrastive Learning for Sentence Representation
• Learning Video Representations using Contrastive Bidirectional Transformer
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• Self-Supervised MultiModal Versatile Networks
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Contrastive Learning for Unpaired Image-to-Image Translation
• Structure-Preserving Image Super-Resolution
• Space-Time Correspondence as a Contrastive Random Walk
• A Comprehensive Study of Deep Video Action Recognition
• Supervised Contrastive Learning
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• A Self-Supervised Gait Encoding Approach with Locality-Awareness for 3D Skeleton Based Person Re-Identification
• Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding
• What Should Not Be Contrastive in Contrastive Learning
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Learning Predictive Representations for Deformable Objects Using Contrastive Estimation
• What makes instance discrimination good for transfer learning?
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Domain Generalization using Causal Matching
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• SEED: Self-supervised Distillation For Visual Representation
• Robust Pre-Training by Adversarial Contrastive Learning
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Contrastive Representation Distillation
• Intriguing Properties of Contrastive Losses
• Video Representation Learning with Visual Tempo Consistency
• Cluster Contrast for Unsupervised Person Re-Identification
• CrossTransformers: spatially-aware few-shot transfer
• Are all negatives created equal in contrastive instance discrimination?
• CoMIR: Contrastive Multimodal Image Representation for Registration
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Contrastive Learning with Adversarial Examples
• Object DGCNN: 3D Object Detection using Dynamic Graphs
• On Mutual Information in Contrastive Learning for Visual Representations
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Trusted Multi-View Classification
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Active Contrastive Learning of Audio-Visual Video Representations
• Contrastive Learning for Recommender System
• Deep Robust Clustering by Contrastive Learning
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation
• Representation Learning via Invariant Causal Mechanisms
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Contrastive Learning with Stronger Augmentations
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Blind Image Super-Resolution via Contrastive Representation Learning
• P4Contrast: Contrastive Learning with Pairs of Point-Pixel Pairs for RGB-D Scene Understanding
• Understanding Self-supervised Learning with Dual Deep Networks
• Contrastive Learning of Musical Representations
• Learning Spatiotemporal Features via Video and Text Pair Discrimination
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Pruning Convolutional Neural Networks with Self-Supervision
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
• Can Temporal Information Help with Contrastive Self-Supervised Learning?
• Self-Supervised Relational Reasoning for Representation Learning
• Contrastive Learning Inverts the Data Generating Process
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Self-Supervised Learning For Few-Shot Image Classification
• Video Understanding as Machine Translation
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• GraphCL: Contrastive Self-Supervised Learning of Graph Representations
• Incremental False Negative Detection for Contrastive Learning
• Understanding the Behaviour of Contrastive Loss
• Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
• Unsupervised Degradation Representation Learning for Blind Super-Resolution
• Deep Reinforcement and InfoMax Learning
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Viewmaker Networks: Learning Views for Unsupervised Representation Learning
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Joint Contrastive Learning for Unsupervised Domain Adaptation
• Unsupervised Representation Learning by InvariancePropagation
• Towards the Generalization of Contrastive Self-Supervised Learning
• Model-Contrastive Federated Learning
• 3D Human Action Representation Learning via Cross-View Consistency Pursuit
• Poisoning and Backdooring Contrastive Learning
• LoCo: Local Contrastive Representation Learning
• Contrastive Syn-to-Real Generalization
• Quasi-Dense Similarity Learning for Multiple Object Tracking
• Contrastive Visual-Linguistic Pretraining
• Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition
• Multimodal Contrastive Training for Visual Representation Learning
• The Next Big Thing(s) in Unsupervised Machine Learning: Five Lessons from Infant Learning
• Debiased Contrastive Learning
• Neural Feature Search for RGB-Infrared Person Re-Identification
• Contrastive Learning for Compact Single Image Dehazing
• DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
• Contrastive learning, multi-view redundancy, and linear models
• Learning Video Representations from Textual Web Supervision
• Momentum Contrastive Voxel-wise Representation Learning for Semi-supervised Volumetric Medical Image Segmentation
• Generalization in Reinforcement Learning by Soft Data Augmentation
• Deep Metric Learning with Spherical Embedding
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Self-supervised Pre-training with Hard Examples Improves Visual Representations
• Prototypical Cross-domain Self-supervised Learning for Few-shot Unsupervised Domain Adaptation
• Self-Damaging Contrastive Learning
• Self-supervised Remote Sensing Images Change Detection at Pixel-level
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• Bi-tuning of Pre-trained Representations
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• Open-Set Hypothesis Transfer with Semantic Consistency
• Self-supervised Modal and View Invariant Feature Learning
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
• Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds
• Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning
• Contrastive Registration for Unsupervised Medical Image Segmentation
• Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning
• Self-supervised Neural Architecture Search
• Joint Contrastive Learning with Infinite Possibilities
• Distilling Localization for Self-Supervised Representation Learning
• Exploring Instance Relations for Unsupervised Feature Embedding
• Unsupervised Learning from Video with Deep Neural Embeddings
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• Self-supervised Learning from a Multi-view Perspective
• Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• Towards Good Practices in Self-supervised Representation Learning
• Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• Self-Paced Contrastive Learning for Semi-supervised Medical Image Segmentation with Meta-labels
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• Category Contrast for Unsupervised Domain Adaptation in Visual Tasks
• MiCE: Mixture of Contrastive Experts for Unsupervised Image Clustering
• Deep Clustering by Semantic Contrastive Learning
• Improving Transformation Invariance in Contrastive Representation Learning
• Multi-label Contrastive Predictive Coding
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Towards Domain-Agnostic Contrastive Learning
• Improving Transferability of Representations via Augmentation-Aware Self-Supervision
• Gradient Regularized Contrastive Learning for Continual Domain Adaptation
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models
• Contrastive Prototype Learning with Augmented Embeddings for Few-Shot Learning
• Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting
• Temporal Context Aggregation for Video Retrieval with Contrastive Learning
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• 3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations
• Unsupervised Object-Level Representation Learning from Scene Images
• QAInfomax: Learning Robust Question Answering System by Mutual Information Maximization
• Online Deep Clustering for Unsupervised Representation Learning
• RetCL: A Selection-based Approach for Retrosynthesis via Contrastive Learning
• For self-supervised learning, Rationality implies generalization, provably
• Generative Models as a Data Source for Multiview Representation Learning
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• Plan2Vec: Unsupervised Representation Learning by Latent Plans
• Few-Shot Electronic Health Record Coding through Graph Contrastive Learning
• DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation
• Calibrating and Improving Graph Contrastive Learning
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• Positional Contrastive Learning for Volumetric Medical Image Segmentation
• Neural Methods for Point-wise Dependency Estimation
• CoKe: Localized Contrastive Learning for Robust Keypoint Detection
• The Role of Global Labels in Few-Shot Classification and How to Infer Them
• Are Labels Necessary for Neural Architecture Search?
• DEMI: Discriminative Estimator of Mutual Information
• SimTriplet: Simple Triplet Representation Learning with a Single GPU
• Progressive Cluster Purification for Unsupervised Feature Learning
• Towards a Hypothesis on Visual Transformation based Self-Supervision
• CLCC: Contrastive Learning for Color Constancy
• Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• Self-Supervised Ranking for Representation Learning
• Temporal Query Networks for Fine-grained Video Understanding
• 3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning
• Rethinking Sampling Strategies for Unsupervised Person Re-identification
• Simple Distillation Baselines for Improving Small Self-supervised Models
• 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• Center-wise Local Image Mixture For Contrastive Representation Learning
• Focus of Attention Improves Information Transfer in Visual Features
• A Simple Framework for Uncertainty in Contrastive Learning
• Quantifying Learnability and Describability of Visual Concepts Emerging in Representation Learning
• Function Contrastive Learning of Transferable Meta-Representations
• Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision
• Self-supervised Representation Learning with Relative Predictive Coding
• Can Pretext-Based Self-Supervised Learning Be Boosted by Downstream Data? A Theoretical Analysis
• Generative Hierarchical Features from Synthesizing Images
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration
• Improving Few-Shot Learning with Auxiliary Self-Supervised Pretext Tasks
• Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting
• Jigsaw Clustering for Unsupervised Visual Representation Learning
• Learning to Track Instances without Video Annotations
• Modern Hopfield Networks for Few- and Zero-Shot Reaction Template Prediction
• StackMix: A complementary Mix algorithm
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Towards Demystifying Representation Learning with Non-contrastive Self-supervision
• Motion-aware Contrastive Video Representation Learning via Foreground-background Merging
• Self-supervised Change Detection in Multi-view Remote Sensing Images
• Data Transformation Insights in Self-supervision with Clustering Tasks
• Self-supervised Feature Learning via Exploiting Multi-modal Data for Retinal Disease Diagnosis
• Knowledge Distillation Meets Self-Supervision
• It Is Likely That Your Loss Should be a Likelihood
• Unsupervised Image Classification for Deep Representation Learning
• Towards DeepSentinel: An extensible corpus of labelled Sentinel-1 and -2 imagery and a general-purpose sensor-fusion semantic embedding model
• Hypothesis Disparity Regularized Mutual Information Maximization
• Aggregative Self-Supervised Feature Learning from a Limited Sample
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Representation Learning for Remote Sensing: An Unsupervised Sensor Fusion Approach
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
• Robust Representation Learning via Perceptual Similarity Metrics
• A Style and Semantic Memory Mechanism for Domain Generalization
• Compressive Visual Representations
• Disentangled Contrastive Learning for Learning Robust Textual Representations
• Contrastive Learning of Global-Local Video Representations
• Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
• Tasting the cake: evaluating self-supervised generalization on out-of-distribution multimodal MRI data
• Self-Supervised Longitudinal Neighbourhood Embedding
• Addressing Feature Suppression in Unsupervised Visual Representations
• A Unified Mixture-View Framework for Unsupervised Representation Learning
• CLOUD: Contrastive Learning of Unsupervised Dynamics
• Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Uniform Priors for Data-Efficient Transfer
• An end-to-end approach for the verification problem: learning the right distance
• Momentum Contrast Speaker Representation Learning
• Contrastive Learning of Relative Position Regression for One-Shot Object Localization in 3D Medical Images
• You Never Cluster Alone
• Information Theory-Guided Heuristic Progressive Multi-View Coding
• Object-Aware Cropping for Self-Supervised Learning
• IDEAL: Independent Domain Embedding Augmentation Learning
• On Contrastive Representations of Stochastic Processes
• Multi-dataset Pretraining: A Unified Model for Semantic Segmentation
• Progressive Stage-wise Learning for Unsupervised Feature Representation Enhancement
• TVDIM: Enhancing Image Self-Supervised Pretraining via Noisy Text Data
• DTG-Net: Differentiated Teachers Guided Self-Supervised Video Action Recognition
• Trainable Class Prototypes for Few-Shot Learning
• Video Contrastive Learning with Global Context
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• Feature Distillation With Guided Adversarial Contrastive Learning
• Resource and data efficient self supervised learning
• Contrastive Video-Language Segmentation
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model
• Unsupervised Object Detection with LiDAR Clues
• On the Marginal Benefit of Active Learning: Does Self-Supervision Eat Its Cake?
• Few Shot Learning With No Labels
• Contrastive Video Representation Learning via Adversarial Perturbations
• Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning
• Manipulator-Independent Representations for Visual Imitation
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• Complementary Relation Contrastive Distillation
• Unsupervised Person Re-identification via Simultaneous Clustering and Consistency Learning
• Interest-Behaviour Multiplicative Network for Resource-limited Recommendation
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Annotation-Efficient Untrimmed Video Action Recognition
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations
• CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning
• Self-supervised SAR-optical Data Fusion and Land-cover Mapping using Sentinel-1/-2 Images
• Electrocardio Panorama: Synthesizing New ECG Views with Self-supervision
• ViTAS: Vision Transformer Architecture Search
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
• Unified Framework for Feature Extraction based on Contrastive Learning
• 6-DoF Contrastive Grasp Proposal Network
• Super-Selfish: Self-Supervised Learning on Images with PyTorch
• Cross-modal Image Retrieval with Deep Mutual Information Maximization
• Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering
• Self Supervision to Distillation for Long-Tailed Visual Recognition
• Towards Unsupervised Sketch-based Image Retrieval
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• Improving Multimodal fusion via Mutual Dependency Maximisation
• Long Short View Feature Decomposition via Contrastive Video Representation Learning
• Cost-Efficient Online Hyperparameter Optimization
• MIC: Model-agnostic Integrated Cross-channel Recommenders
• Pedestrian Detection by Exemplar-Guided Contrastive Learning
• An Image-based Approach of Task-driven Driving Scene Categorization
• Voxel-level Siamese Representation Learning for Abdominal Multi-Organ Segmentation
• Unsupervised Few-Shot Action Recognition via Action-Appearance Aligned Meta-Adaptation
• Graph Partner Neural Networks for Semi-Supervised Learning on Graphs
• Inverse Problems Leveraging Pre-trained Contrastive Representations
• Meta-Learning to Improve Pre-Training
• Recent Advancements in Self-Supervised Paradigms for Visual Feature Representation
• Feature Lenses: Plug-and-play Neural Modules for Transformation-Invariant Visual Representations
• On Compositions of Transformations in Contrastive Self-Supervised Learning
• Leveraging Hidden Structure in Self-Supervised Learning
• What Remains of Visual Semantic Embeddings
• Boosting Few-Shot Classification with View-Learnable Contrastive Learning
• SVMax: A Feature Embedding Regularizer
• PCPs: Patient Cardiac Prototypes