Hard Negative Mixing for Contrastive Learning

Abstract

Contrastive learning has become a key component of self-supervised learning approaches for computer vision. By learning to embed two augmented versions of the same image close to each other and to push the embeddings of different images apart, one can train highly transferable visual representations. As revealed by recent studies, heavy data augmentation and large sets of negatives are both crucial in learning such representations. At the same time, data mixing strategies either at the image or the feature level improve both supervised and semi-supervised learning by synthesizing novel examples, forcing networks to learn more robust features. In this paper, we argue that an important aspect of contrastive learning, i.e., the effect of hard negatives, has so far been neglected. To get more meaningful negative samples, current top contrastive self-supervised learning approaches either substantially increase the batch sizes, or keep very large memory banks; increasing the memory size, however, leads to diminishing returns in terms of performance. We therefore start by delving deeper into a top-performing framework and show evidence that harder negatives are needed to facilitate better and faster learning. Based on these observations, and motivated by the success of data mixing, we propose hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead. We exhaustively ablate our approach on linear classification, object detection and instance segmentation and show that employing our hard negative mixing procedure improves the quality of visual representations learned by a state-of-the-art self-supervised learning method.

arXiv commentAccepted at NeurIPS 2020. Project page with pretrained models: https://europe.naverlabs.com/mochi

References in corpus (34)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Improved Baselines with Momentum Contrastive Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Learning of Video Representations using LSTMs
• Interpolation Consistency Training for Semi-Supervised Learning
• Learning Representations by Maximizing Mutual Information Across Views
• What Makes for Good Views for Contrastive Learning?
• Contrastive Multiview Coding
• Prototypical Contrastive Learning of Unsupervised Representations
• Manifold Mixup: Better Representations by Interpolating Hidden States
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Exploring Simple Siamese Representation Learning
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• On Mutual Information Maximization for Representation Learning
• Learning Video Representations using Contrastive Bidirectional Transformer
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• A critical analysis of self-supervision, or what we can learn from a single image
• What makes instance discrimination good for transfer learning?
• Labelling unlabelled videos from scratch with multi-modal self-supervision
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• A Framework For Contrastive Self-Supervised Learning And Designing A New Approach
• Self-supervised Video Representation Learning by Pace Prediction
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• On Mutual Information in Contrastive Learning for Visual Representations
• Predicting What You Already Know Helps: Provable Self-Supervised Learning
• Deep Robust Clustering by Contrastive Learning
• Understanding Self-supervised Learning with Dual Deep Networks
• Hybrid Discriminative-Generative Training via Contrastive Learning
• LoCo: Local Contrastive Representation Learning
• Debiased Contrastive Learning
• CO2: Consistent Contrast for Unsupervised Visual Representation Learning
• K-Shot Contrastive Learning of Visual Features with Multiple Instance Augmentations

Cited by in corpus (27)

• Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• Augmented Negative Sampling for Collaborative Filtering
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Multi-queue Momentum Contrast for Microvideo-Product Retrieval
• Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learning
• Universum-inspired Supervised Contrastive Learning
• Supervised Contrastive Learning with Hard Negative Samples
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample Prediction
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries
• Understand and Improve Contrastive Learning Methods for Visual Representation: A Review
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• It Takes Two to Tango: Mixup for Deep Metric Learning
• Contrastive Semi-Supervised Learning for 2D Medical Image Segmentation
• A Survey of Latent Factor Models in Recommender Systems
• Supporting Clustering with Contrastive Learning
• Understanding Hyperbolic Metric Learning through Hard Negative Sampling
• Contrastive Learning and Cycle Consistency-based Transductive Transfer Learning for Target Annotation
• Causal Unsupervised Semantic Segmentation
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Contrastive Learning for Self-Supervised Pre-Training of Point Cloud Segmentation Networks With Image Data
• UserBERT: Contrastive User Model Pre-training
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Temporal Knowledge Consistency for Unsupervised Visual Representation Learning
• Progressive Multi-stage Feature Mix for Person Re-Identification