Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss

Abstract

Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes. We design two novel methods to improve performance in such scenarios. First, we propose a theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound. This loss replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Second, we propose a simple, yet effective, training schedule that defers re-weighting until after the initial stage, allowing the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling. We test our methods on several benchmark vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our experiments show that either of these methods alone can already improve over existing techniques and their combination achieves even better performance gains.

arXiv commentto appear in NeurIPS 2019

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• SMOTE: Synthetic Minority Over-sampling Technique
• A systematic study of the class imbalance problem in convolutional neural networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Deep Visual Domain Adaptation: A Survey
• YFCC100M: The New Data in Multimedia Research
• Fairness Without Demographics in Repeated Loss Minimization
• The Devil is in the Tails: Fine-grained Classification in the Wild
• Rethinking Feature Discrimination and Polymerization for Large-scale Recognition
• Stronger generalization bounds for deep nets via a compression approach
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Gradient Harmonized Single-stage Detector
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• Regularized Learning for Domain Adaptation under Label Shifts
• Domain Adaptation for Semantic Segmentation via Class-Balanced Self-Training
• Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Variance Reduction for Matrix Games
• Evaluating Fairness Metrics in the Presence of Dataset Bias

Cited by in corpus (31)

• A Survey on Long-Tailed Visual Recognition
• Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training
• Feature Fusion from Head to Tail for Long-Tailed Visual Recognition
• Class-Difficulty Based Methods for Long-Tailed Visual Recognition
• Class-Imbalanced Semi-Supervised Learning
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Towards Balanced Learning for Instance Recognition
• Distribution-aware Margin Calibration for Semantic Segmentation in Images
• Jointly De-biasing Face Recognition and Demographic Attribute Estimation
• Self-Damaging Contrastive Learning
• Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning
• Deep Extrapolation for Attribute-Enhanced Generation
• ACE: Ally Complementary Experts for Solving Long-Tailed Recognition in One-Shot
• Multi-label Contrastive Predictive Coding
• Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
• MSE Loss with Outlying Label for Imbalanced Classification
• Overcoming Classifier Imbalance for Long-tail Object Detection with Balanced Group Softmax
• Semi-supervised Long-tailed Recognition using Alternate Sampling
• The Devil is in Classification: A Simple Framework for Long-tail Object Detection and Instance Segmentation
• Graph Classification by Mixture of Diverse Experts
• FedSC: Federated Learning with Semantic-Aware Collaboration
• Investigate the Essence of Long-Tailed Recognition from a Unified Perspective
• Adjusting Decision Boundary for Class Imbalanced Learning
• Hierarchical Knowledge Guided Learning for Real-world Retinal Diseases Recognition
• Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition
• Learning with Multiclass AUC: Theory and Algorithms
• The Best of Both Worlds: a Framework for Combining Degradation Prediction with High Performance Super-Resolution Networks
• A Linkage-based Doubly Imbalanced Graph Learning Framework for Face Clustering
• BiSTF: Bilateral-Branch Self-Training Framework for Semi-Supervised Large-scale Fine-Grained Recognition
• On-target Adaptation