Training Over-parameterized Models with Non-decomposable Objectives

Abstract

Many modern machine learning applications come with complex and nuanced design goals such as minimizing the worst-case error, satisfying a given precision or recall target, or enforcing group-fairness constraints. Popular techniques for optimizing such non-decomposable objectives reduce the problem into a sequence of cost-sensitive learning tasks, each of which is then solved by re-weighting the training loss with example-specific costs. We point out that the standard approach of re-weighting the loss to incorporate label costs can produce unsatisfactory results when used to train over-parameterized models. As a remedy, we propose new cost-sensitive losses that extend the classical idea of logit adjustment to handle more general cost matrices. Our losses are calibrated, and can be further improved with distilled labels from a teacher model. Through experiments on benchmark image datasets, we showcase the effectiveness of our approach in training ResNet models with common robust and constrained optimization objectives.

References in corpus (22)

• SMOTE: Synthetic Minority Over-sampling Technique
• Distilling the Knowledge in a Neural Network
• A systematic study of the class imbalance problem in convolutional neural networks
• Equality of Opportunity in Supervised Learning
• The Devil is in the Tails: Fine-grained Classification in the Wild
• Class-Balanced Loss Based on Effective Number of Samples
• Satisfying Real-world Goals with Dataset Constraints
• Optimizing F-measure: A Tale of Two Approaches
• Policy Distillation
• Training Deep Neural Networks via Direct Loss Minimization
• No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems
• Distribution Alignment: A Unified Framework for Long-tail Visual Recognition
• Addressing the Loss-Metric Mismatch with Adaptive Loss Alignment
• Optimizing Non-decomposable Performance Measures: A Tale of Two Classes
• Online and Stochastic Gradient Methods for Non-decomposable Loss Functions
• Convex Calibration Dimension for Multiclass Loss Matrices
• To Balance or Not to Balance: A Simple-yet-Effective Approach for Learning with Long-Tailed Distributions
• Balanced Knowledge Distillation for Long-tailed Learning
• Binary Classification with Karmic, Threshold-Quasi-Concave Metrics
• Label-Imbalanced and Group-Sensitive Classification under Overparameterization
• Reviving Threshold-Moving: a Simple Plug-in Bagging Ensemble for Binary and Multiclass Imbalanced Data
• Implicit Rate-Constrained Optimization of Non-decomposable Objectives