Contrastive Learning with Hard Negative Samples

Abstract

How can you sample good negative examples for contrastive learning? We argue that, as with metric learning, contrastive learning of representations benefits from hard negative samples (i.e., points that are difficult to distinguish from an anchor point). The key challenge toward using hard negatives is that contrastive methods must remain unsupervised, making it infeasible to adopt existing negative sampling strategies that use true similarity information. In response, we develop a new family of unsupervised sampling methods for selecting hard negative samples where the user can control the hardness. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.

arXiv commentPublished as a conference paper at ICLR 2021

References in corpus (11)

• Improved Baselines with Momentum Contrastive Learning
• A Survey on Multi-view Learning
• Skip-Thought Vectors
• What Makes for Good Views for Contrastive Learning?
• An efficient framework for learning sentence representations
• Contrastive Multi-View Representation Learning on Graphs
• TUDataset: A collection of benchmark datasets for learning with graphs
• Graph Matching Networks for Learning the Similarity of Graph Structured Objects
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
• Estimating the class prior and posterior from noisy positives and unlabeled data
• Locality and compositionality in zero-shot learning

Cited by in corpus (11)

• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• Contrastive Learning Inverts the Data Generating Process
• Improving Text-to-Image Synthesis Using Contrastive Learning
• Self-supervised EEG Representation Learning for Automatic Sleep Staging
• Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE
• Understanding and Achieving Efficient Robustness with Adversarial Supervised Contrastive Learning
• UserBERT: Contrastive User Model Pre-training
• Self-supervised Change Detection in Multi-view Remote Sensing Images
• Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning
• Information Theory-Guided Heuristic Progressive Multi-View Coding
• Learning with partially separable data