Hard Negative Sampling via Regularized Optimal Transport for Contrastive Representation Learning

Abstract

We study the problem of designing hard negative sampling distributions for unsupervised contrastive representation learning. We propose and analyze a novel min-max framework that seeks a representation which minimizes the maximum (worst-case) generalized contrastive learning loss over all couplings (joint distributions between positive and negative samples subject to marginal constraints) and prove that the resulting min-max optimum representation will be degenerate. This provides the first theoretical justification for incorporating additional regularization constraints on the couplings. We re-interpret the min-max problem through the lens of Optimal Transport (OT) theory and utilize regularized transport couplings to control the degree of hardness of negative examples. Through experiments we demonstrate that the negative samples generated from our designed negative distribution are more similar to the anchor than those generated from the baseline negative distribution. We also demonstrate that entropic regularization yields negative sampling distributions with parametric form similar to that in a recent state-of-the-art negative sampling design and has similar performance in multiple datasets. Utilizing the uncovered connection with OT, we propose a new ground cost for designing the negative distribution and show improved performance of the learned representation on downstream tasks compared to the representation learned when using squared Euclidean cost.

Minor changes - fixing typos, minor rephrasing in abstract and introduction

References in corpus (10)

• Deep Graph Contrastive Representation Learning
• Contrastive Multi-View Representation Learning on Graphs
• TUDataset: A collection of benchmark datasets for learning with graphs
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Are all negatives created equal in contrastive instance discrimination?
• Contrastive Learning with Hard Negative Samples
• Conditional Negative Sampling for Contrastive Learning of Visual Representations
• Debiased Contrastive Learning
• Representation Learning via Adversarially-Contrastive Optimal Transport
• A Tale of Two Efficient and Informative Negative Sampling Distributions