Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
arXiv:2105.05736
Abstract
Negative sampling schemes enable efficient training given a large number of classes, by offering a means to approximate a computationally expensive loss function that takes all labels into account. In this paper, we present a new connection between these schemes and loss modification techniques for countering label imbalance. We show that different negative sampling schemes implicitly trade-off performance on dominant versus rare labels. Further, we provide a unified means to explicitly tackle both sampling bias, arising from working with a subset of all labels, and labeling bias, which is inherent to the data due to label imbalance. We empirically verify our findings on long-tail classification and retrieval benchmarks.
arXiv commentTo appear in ICML 2021
References in corpus (13)
- SMOTE: Synthetic Minority Over-sampling Technique
- Distributed Representations of Words and Phrases and their Compositionality
- A systematic study of the class imbalance problem in convolutional neural networks
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- The Devil is in the Tails: Fine-grained Classification in the Wild
- Class-Balanced Loss Based on Effective Number of Samples
- Variance Reduction in SGD by Distributed Importance Sampling
- Long-tail learning via logit adjustment
- A no-regret generalization of hierarchical softmax to extreme multi-label classification
- Stochastic Negative Mining for Learning with Large Output Spaces
- Equalization Loss for Long-Tailed Object Recognition
- Extreme Classification via Adversarial Softmax Approximation