Data-Efficient Image Recognition with Contrastive Predictive Coding

Abstract

Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-the-art linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5x less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained ImageNet classifiers.

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• Representation Learning with Contrastive Predictive Coding
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• The information bottleneck method
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Data Augmentation for Consistency Training
• YFCC100M: The New Data in Multimedia Research
• Semi-Supervised Learning with Deep Generative Models
• Going Deeper with Convolutions
• Momentum Contrast for Unsupervised Visual Representation Learning
• AutoAugment: Learning Augmentation Policies from Data
• Adversarial Feature Learning
• Learning Representations by Maximizing Mutual Information Across Views
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Contrastive Multiview Coding
• Layer Normalization
• Large Scale Adversarial Representation Learning
• Good Semi-supervised Learning that Requires a Bad GAN
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Learning to Poke by Poking: Experiential Learning of Intuitive Physics
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Generic 3D Representation via Pose Estimation and Matching
• Revisiting Self-Supervised Visual Representation Learning
• Time-Contrastive Networks: Self-Supervised Learning from Video

Cited by in corpus (311)

• A Simple Framework for Contrastive Learning of Visual Representations
• Learning Transferable Visual Models From Natural Language Supervision
• Transformers in Vision: A Survey
• Bootstrap your own latent: A new approach to self-supervised Learning
• Learning to Prompt for Vision-Language Models
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Unsupervised Data Augmentation for Consistency Training
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Graph Contrastive Learning with Adaptive Augmentation
• DINOv2: Learning Robust Visual Features without Supervision
• Momentum Contrast for Unsupervised Visual Representation Learning
• Contrastive Representation Learning: A Framework and Review
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
• Exploring Simple Siamese Representation Learning
• Deep Graph Contrastive Representation Learning
• Large Scale Adversarial Representation Learning
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
• On Mutual Information Maximization for Representation Learning
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances
• Self-supervised Co-training for Video Representation Learning
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Reinforcement Learning with Augmented Data
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion
• Learning Video Representations using Contrastive Bidirectional Transformer
• A survey on Semi-, Self- and Unsupervised Learning for Image Classification
• CURL: Contrastive Unsupervised Representations for Reinforcement Learning
• Rethinking the Value of Labels for Improving Class-Imbalanced Learning
• Self-Supervised MultiModal Versatile Networks
• Momentum Contrastive Learning for Few-Shot COVID-19 Diagnosis from Chest CT Images
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Self-supervised Pretraining of Visual Features in the Wild
• Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• 3D Self-Supervised Methods for Medical Imaging
• Contrastive Learning for Unpaired Image-to-Image Translation
• Reinventing 2D Convolutions for 3D Images
• Supervised Contrastive Learning
• Space-Time Correspondence as a Contrastive Random Walk
• Self-labelling via simultaneous clustering and representation learning
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding
• Learning and Evaluating Representations for Deep One-class Classification
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• What makes instance discrimination good for transfer learning?
• Contrastive Collaborative Filtering for Cold-Start Item Recommendation
• Spatiotemporal Contrastive Video Representation Learning
• Memory-augmented Dense Predictive Coding for Video Representation Learning
• Self-supervised deep convolutional neural network for chest X-ray classification
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• Selfie: Self-supervised Pretraining for Image Embedding
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• SEED: Self-supervised Distillation For Visual Representation
• Are we done with ImageNet?
• Robust Pre-Training by Adversarial Contrastive Learning
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Contrastive Learning of Structured World Models
• Decoupling Representation Learning from Reinforcement Learning
• Intriguing Properties of Contrastive Losses
• Self-Supervised Policy Adaptation during Deployment
• Video Representation Learning with Visual Tempo Consistency
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Cluster Contrast for Unsupervised Person Re-Identification
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• Exploring Contrastive Learning in Human Activity Recognition for Healthcare
• CoMIR: Contrastive Multimodal Image Representation for Registration
• Aligning Pretraining for Detection via Object-Level Contrastive Learning
• i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning
• Self-Regulated Learning for Egocentric Video Activity Anticipation
• MixCo: Mix-up Contrastive Learning for Visual Representation
• Contrastive Training for Improved Out-of-Distribution Detection
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Active Contrastive Learning of Audio-Visual Video Representations
• A neural network walks into a lab: towards using deep nets as models for human behavior
• Imitating Interactive Intelligence
• Deep Robust Clustering by Contrastive Learning
• Whitening for Self-Supervised Representation Learning
• Automatic Shortcut Removal for Self-Supervised Representation Learning
• Detecting Outliers with Foreign Patch Interpolation
• Representation Learning via Invariant Causal Mechanisms
• Contrastive Knowledge Graph Error Detection
• Watching the World Go By: Representation Learning from Unlabeled Videos
• Contrastive Learning with Stronger Augmentations
• Contrastive Self-supervised Neural Architecture Search
• Contrastive Learning of Musical Representations
• Semi-Supervised Histology Classification using Deep Multiple Instance Learning and Contrastive Predictive Coding
• Pretraining Representations for Data-Efficient Reinforcement Learning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Data-Efficient Instance Generation from Instance Discrimination
• Pruning Convolutional Neural Networks with Self-Supervision
• Multi-Format Contrastive Learning of Audio Representations
• CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP
• MTS-DVGAN: Anomaly Detection in Cyber-Physical Systems using a Dual Variational Generative Adversarial Network
• Contrastive Learning Inverts the Data Generating Process
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• False: False Negative Samples Aware Contrastive Learning for Semantic Segmentation of High-Resolution Remote Sensing Image
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• Contrastive Multiview Coding with Electro-optics for SAR Semantic Segmentation
• Self-Supervised Learning For Few-Shot Image Classification
• Rethinking "Batch" in BatchNorm
• Understanding the Behaviour of Contrastive Loss
• Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretraining
• Deep Reinforcement and InfoMax Learning
• Self-supervised Video Representation Learning by Uncovering Spatio-temporal Statistics
• Unsupervised Degradation Representation Learning for Blind Super-Resolution
• Self-Supervised Learning with Kernel Dependence Maximization
• Self-Supervised Variational Auto-Encoders
• Unsupervised Representation Learning by InvariancePropagation
• Multimodal Self-Supervised Learning of General Audio Representations
• A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
• Contrastive Learning for Fair Representations
• One Epoch Is All You Need
• Quasi-Dense Similarity Learning for Multiple Object Tracking
• Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Contrastive Attention for Automatic Chest X-ray Report Generation
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• Contrastive Syn-to-Real Generalization
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Debiased Contrastive Learning
• Multimodal Contrastive Training for Visual Representation Learning
• Local plasticity rules can learn deep representations using self-supervised contrastive predictions
• Deeply Unsupervised Patch Re-Identification for Pre-training Object Detectors
• Contrastive Learning for Compact Single Image Dehazing
• Self-supervised Pre-training with Hard Examples Improves Visual Representations
• An Asymmetric Contrastive Loss for Handling Imbalanced Datasets
• On Linear Identifiability of Learned Representations
• Motif-Driven Contrastive Learning of Graph Representations
• Joint Learning of Localized Representations from Medical Images and Reports
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Dual Contrastive Learning for Unsupervised Image-to-Image Translation
• When Does Self-supervision Improve Few-shot Learning?
• SimMIM: A Simple Framework for Masked Image Modeling
• CLAR: Contrastive Learning of Auditory Representations
• Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning
• Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation
• Small-Text: Active Learning for Text Classification in Python
• Contrastive Predictive Coding for Anomaly Detection
• Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning
• Exploring wav2vec 2.0 on speaker verification and language identification
• Elastic-InfoGAN: Unsupervised Disentangled Representation Learning in Class-Imbalanced Data
• Learning Robust and Multilingual Speech Representations
• Unsupervised Feature Learning by Cross-Level Instance-Group Discrimination
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Global-Local Bidirectional Reasoning for Unsupervised Representation Learning of 3D Point Clouds
• Self-supervised Modal and View Invariant Feature Learning
• Joint Contrastive Learning with Infinite Possibilities
• Self-supervised Learning from a Multi-view Perspective
• Doubly Contrastive Deep Clustering
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
• Unsupervised Deep Metric Learning via Auxiliary Rotation Loss
• Unsupervised Learning from Video with Deep Neural Embeddings
• Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning
• How Useful is Self-Supervised Pretraining for Visual Tasks?
• The Spatially-Correlative Loss for Various Image Translation Tasks
• Analysis of Predictive Coding Models for Phonemic Representation Learning in Small Datasets
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• Training Larger Networks for Deep Reinforcement Learning
• Category Contrast for Unsupervised Domain Adaptation in Visual Tasks
• Spatially Consistent Representation Learning
• Multi-label Contrastive Predictive Coding
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Probing the State of the Art: A Critical Look at Visual Representation Evaluation
• Towards Domain-Agnostic Contrastive Learning
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• Evaluating representations by the complexity of learning low-loss predictors
• Improving Transformation Invariance in Contrastive Representation Learning
• Parallel Training of Deep Networks with Local Updates
• Adversarial Training with Contrastive Learning in NLP
• Training GANs with Stronger Augmentations via Contrastive Discriminator
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Self-Supervised Learning Aided Class-Incremental Lifelong Learning
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
• MixSiam: A Mixture-based Approach to Self-supervised Representation Learning
• Self-supervised Semi-supervised Learning for Data Labeling and Quality Evaluation
• Online Deep Clustering for Unsupervised Representation Learning
• Mutual Information Maximization for Robust Plannable Representations
• RetCL: A Selection-based Approach for Retrosynthesis via Contrastive Learning
• Semantic-Aware Generation for Self-Supervised Visual Representation Learning
• OpenCoS: Contrastive Semi-supervised Learning for Handling Open-set Unlabeled Data
• Language-Driven Image Style Transfer
• Unsupervised Object-Level Representation Learning from Scene Images
• Generative Models as a Data Source for Multiview Representation Learning
• Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning
• CoKe: Localized Contrastive Learning for Robust Keypoint Detection
• Neural Methods for Point-wise Dependency Estimation
• Calibrating and Improving Graph Contrastive Learning
• Auxiliary Tasks Speed Up Learning PointGoal Navigation
• PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapse
• Representation Learning via Consistent Assignment of Views to Clusters
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
• CLCC: Contrastive Learning for Color Constancy
• Pneumonia Detection on Chest X-ray using Radiomic Features and Contrastive Learning
• Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning
• DEMI: Discriminative Estimator of Mutual Information
• Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework
• Multi-Task Self-Training for Learning General Representations
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• Towards Deviation-Robust Agent Navigation via Perturbation-Aware Contrastive Learning
• Slot Contrastive Networks: A Contrastive Approach for Representing Objects
• Telescoping Density-Ratio Estimation
• Generative Hierarchical Features from Synthesizing Images
• Semantically-Conditioned Negative Samples for Efficient Contrastive Learning
• An empirical study of pretrained representations for few-shot classification
• Deep Contrastive Graph Representation via Adaptive Homotopy Learning
• COBRA: Contrastive Bi-Modal Representation Algorithm
• Self-Supervised Learning of a Biologically-Inspired Visual Texture Model
• EqCo: Equivalent Rules for Self-supervised Contrastive Learning
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening
• Unsupervised Detection of Distinctive Regions on 3D Shapes
• Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery
• Quantifying Learnability and Describability of Visual Concepts Emerging in Representation Learning
• Center-wise Local Image Mixture For Contrastive Representation Learning
• Contrastive Neural Processes for Self-Supervised Learning
• KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
• Cluster Analysis with Deep Embeddings and Contrastive Learning
• Improving Few-Shot Learning with Auxiliary Self-Supervised Pretext Tasks
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Learning Visual Representations for Transfer Learning by Suppressing Texture
• Unsupervised Part Discovery from Contrastive Reconstruction
• Learnt dynamics generalizes across tasks, datasets, and populations
• Self-supervised Contrastive Video-Speech Representation Learning for Ultrasound
• DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck
• Clustering-Based Representation Learning through Output Translation and Its Application to Remote--Sensing Images
• Jigsaw Clustering for Unsupervised Visual Representation Learning
• Detecting Outliers with Poisson Image Interpolation
• Conditional Mutual information-based Contrastive Loss for Financial Time Series Forecasting
• Learning a Domain-Agnostic Visual Representation for Autonomous Driving via Contrastive Loss
• Practical Assessment of Generalization Performance Robustness for Deep Networks via Contrastive Examples
• On the Evaluation and Real-World Usage Scenarios of Deep Vessel Segmentation for Retinography
• Compressive Visual Representations
• Self-supervised Feature Learning via Exploiting Multi-modal Data for Retinal Disease Diagnosis
• Addressing Feature Suppression in Unsupervised Visual Representations
• Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
• Representation Learning for Remote Sensing: An Unsupervised Sensor Fusion Approach
• Enhancing Sound Source Localization via False Negative Elimination
• Aggregative Self-Supervised Feature Learning from a Limited Sample
• Memory Augmented Multi-Instance Contrastive Predictive Coding for Sequential Recommendation
• Self-Supervised Video Representation Learning with Meta-Contrastive Network
• Watching Too Much Television is Good: Self-Supervised Audio-Visual Representation Learning from Movies and TV Shows
• Pretrained Encoders are All You Need
• Unsupervised Image Classification for Deep Representation Learning
• Functional Regularization for Representation Learning: A Unified Theoretical Perspective
• Contrastive Video Representation Learning via Adversarial Perturbations
• Dissecting Supervised Contrastive Learning
• Contrastive Learning and Self-Training for Unsupervised Domain Adaptation in Semantic Segmentation
• Boosting Semi-Supervised Face Recognition with Noise Robustness
• Information Theory-Guided Heuristic Progressive Multi-View Coding
• ExCon: Explanation-driven Supervised Contrastive Learning for Image Classification
• Improving Image Clustering through Sample Ranking and Its Application to remote--sensing images
• What Can You Learn from Your Muscles? Learning Visual Representation from Human Interactions
• IDEAL: Independent Domain Embedding Augmentation Learning
• Progressive Stage-wise Learning for Unsupervised Feature Representation Enhancement
• Interest-Behaviour Multiplicative Network for Resource-limited Recommendation
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• COIN: Contrastive Identifier Network for Breast Mass Diagnosis in Mammography
• The emergence of visual semantics through communication games
• Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning
• Enabling the Network to Surf the Internet
• Fast Development of ASR in African Languages using Self Supervised Speech Representation Learning
• UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
• Self-Supervised Dynamic Networks for Covariate Shift Robustness
• How Convolutional Neural Network Architecture Biases Learned Opponency and Colour Tuning
• CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning
• Improving speech recognition models with small samples for air traffic control systems
• On Equivariant and Invariant Learning of Object Landmark Representations
• Hybrid BYOL-ViT: Efficient approach to deal with small datasets
• On Compositions of Transformations in Contrastive Self-Supervised Learning
• A Collective Learning Framework to Boost GNN Expressiveness
• Cross-modal Image Retrieval with Deep Mutual Information Maximization
• SupCL-Seq: Supervised Contrastive Learning for Downstream Optimized Sequence Representations
• Synthetic-to-Real Domain Adaptation for Lane Detection
• Improved Detection of Face Presentation Attacks Using Image Decomposition
• Deep Bregman Divergence for Contrastive Learning of Visual Representations
• Leveraging Hidden Structure in Self-Supervised Learning
• Unsupervised Discovery of the Long-Tail in Instance Segmentation Using Hierarchical Self-Supervision
• A Review of Computer Vision Methods in Network Security
• Self-Supervised Learning for Large-Scale Unsupervised Image Clustering
• Voxel-level Siamese Representation Learning for Abdominal Multi-Organ Segmentation
• Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency
• Synthesizing Photorealistic Images with Deep Generative Learning
• Unsupervised Representation Learning via Neural Activation Coding
• A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
• Stabilizing Label Assignment for Speech Separation by Self-supervised Pre-training
• Dual Prototypical Contrastive Learning for Few-shot Semantic Segmentation
• Meta-Learning to Improve Pre-Training
• About contrastive unsupervised representation learning for classification and its convergence
• Super-Selfish: Self-Supervised Learning on Images with PyTorch
• Post-Workshop Report on Science meets Engineering in Deep Learning, NeurIPS 2019, Vancouver
• Contrastive Learning of Visual-Semantic Embeddings
• Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation
• Label-Efficient Multi-Task Segmentation using Contrastive Learning
• Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learning