Stochastic Negative Mining for Learning with Large Output Spaces

Abstract

We consider the problem of retrieving the most relevant labels for a given input when the size of the output space is very large. Retrieval methods are modeled as set-valued classifiers which output a small set of classes for each input, and a mistake is made if the label is not in the output set. Despite its practical importance, a statistically principled, yet practical solution to this problem is largely missing. To this end, we first define a family of surrogate losses and show that they are calibrated and convex under certain conditions on the loss parameters and data distribution, thereby establishing a statistical and analytical basis for using these losses. Furthermore, we identify a particularly intuitive class of loss functions in the aforementioned family and show that they are amenable to practical implementation in the large output space setting (i.e. computation is possible without evaluating scores of all labels) by developing a technique called Stochastic Negative Mining. We also provide generalization error bounds for the losses in the family. Finally, we conduct experiments which demonstrate that Stochastic Negative Mining yields benefits over commonly used negative sampling approaches.

Cited by in corpus (15)

• Federated Learning with Only Positive Labels
• PECOS: Prediction for Enormous and Correlated Output Spaces
• Why distillation helps: a statistical perspective
• Taming Pretrained Transformers for Extreme Multi-label Text Classification
• sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification
• Prototype Memory for Large-scale Face Representation Learning
• Label Disentanglement in Partition-based Extreme Multilabel Classification
• Unbiased Loss Functions for Extreme Classification With Missing Labels
• Optimistic bounds for multi-output prediction
• Doubly-stochastic mining for heterogeneous retrieval
• Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
• Instance-based Transfer Learning for Multilingual Deep Retrieval
• Sampled Softmax with Random Fourier Features
• Improving Calibration in Deep Metric Learning With Cross-Example Softmax
• Extreme Multi-label Learning for Semantic Matching in Product Search