Long-tail learning via logit adjustment

Abstract

Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes naïve learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies, either applied post-hoc to a trained model, or enforced in the loss during training. Such adjustment encourages a large relative margin between logits of rare versus dominant labels. These techniques unify and generalise several recent proposals in the literature, while possessing firmer statistical grounding and empirical performance.

arXiv commentPublished as a conference paper in ICLR 2021

References in corpus (15)

• SMOTE: Synthetic Minority Over-sampling Technique
• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• A systematic study of the class imbalance problem in convolutional neural networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On Calibration of Modern Neural Networks
• When Does Label Smoothing Help?
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Composite Binary Losses
• The Devil is in the Tails: Fine-grained Classification in the Wild
• Class-Balanced Loss Based on Effective Number of Samples
• Learning with Average Top-k Loss
• To Balance or Not to Balance: A Simple-yet-Effective Approach for Learning with Long-Tailed Distributions
• Reviving Threshold-Moving: a Simple Plug-in Bagging Ensemble for Binary and Multiclass Imbalanced Data
• Approximated Structured Prediction for Learning Large Scale Graphical Models

Cited by in corpus (14)

• Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
• CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning
• Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification
• Feature-Balanced Loss for Long-Tailed Visual Recognition
• ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot
• Balanced Knowledge Distillation for Long-tailed Learning
• A Model of Two Tales: Dual Transfer Learning Framework for Improved Long-tail Item Recommendation
• Distilling Virtual Examples for Long-tailed Recognition
• Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning
• Structured Sparse R-CNN for Direct Scene Graph Generation
• Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces
• Post-hoc Models for Performance Estimation of Machine Learning Inference
• Calibrating Class Activation Maps for Long-Tailed Visual Recognition
• A Conditional Cascade Model for Relational Triple Extraction