Label-Imbalanced and Group-Sensitive Classification under Overparameterization
arXiv:2103.01550
Abstract
The goal in label-imbalanced and group-sensitive classification is to optimize relevant metrics such as balanced error and equal opportunity. Classical methods, such as weighted cross-entropy, fail when training deep nets to the terminal phase of training (TPT), that is training beyond zero training error. This observation has motivated recent flurry of activity in developing heuristic alternatives following the intuitive mechanism of promoting larger margin for minorities. In contrast to previous heuristics, we follow a principled analysis explaining how different loss adjustments affect margins. First, we prove that for all linear classifiers trained in TPT, it is necessary to introduce multiplicative, rather than additive, logit adjustments so that the interclass margins change appropriately. To show this, we discover a connection of the multiplicative CE modification to the cost-sensitive support-vector machines. Perhaps counterintuitively, we also find that, at the start of training, the same multiplicative weights can actually harm the minority classes. Thus, while additive adjustments are ineffective in the TPT, we show that they can speed up convergence by countering the initial negative effect of the multiplicative weights. Motivated by these findings, we formulate the vector-scaling (VS) loss, that captures existing techniques as special cases. Moreover, we introduce a natural extension of the VS-loss to group-sensitive classification, thus treating the two common types of imbalances (label/group) in a unifying way. Importantly, our experiments on state-of-the-art datasets are fully consistent with our theoretical insights and confirm the superior performance of our algorithms. Finally, for imbalanced Gaussian-mixtures data, we perform a generalization analysis, revealing tradeoffs between balanced / standard error and equal opportunity.
References in corpus (25)
- Message Passing Algorithms for Compressed Sensing
- Equality of Opportunity in Supervised Learning
- On Calibration of Modern Neural Networks
- Benign Overfitting in Linear Regression
- Prevalence of Neural Collapse during the terminal phase of deep learning training
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- On the (im)possibility of fairness
- The generalization error of random features regression: Precise asymptotics and double descent curve
- A modern maximum-likelihood theory for high-dimensional logistic regression
- Does Distributionally Robust Supervised Learning Give Robust Classifiers?
- A framework to characterize performance of LASSO algorithms
- The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
- An Investigation of Why Overparameterization Exacerbates Spurious Correlations
- Risk and parameter convergence of logistic regression
- Symbol Error Rate Performance of Box-relaxation Decoders in Massive MIMO
- Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?
- Classification vs regression in overparameterized regimes: Does the loss function matter?
- Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
- Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularization
- A Precise Performance Analysis of Learning with Random Features
- Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions
- Characterizing the implicit bias via a primal-dual analysis
- Neural Collapse with Cross-Entropy Loss
- On the Role of Dataset Quality and Heterogeneity in Model Confidence
- Factors in Finetuning Deep Model for object detection