Selective Classification Can Magnify Disparities Across Groups

Abstract

Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations. We observe this behavior consistently across five vision and NLP datasets. Surprisingly, increasing abstentions can even decrease accuracies on some groups. To better understand this phenomenon, we study the margin distribution, which captures the model's confidences over all predictions. For symmetric margin distributions, we prove that whether selective classification monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisfies a property we call left-log-concavity. Our analysis also shows that selective classification tends to magnify full-coverage accuracy disparities. Motivated by our analysis, we train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group on these models. Altogether, our results suggest that selective classification should be used with care and underscore the importance of training models to perform equally well across groups at full coverage.

arXiv commentPublished at the International Conference on Learning Representations (ICLR) 2021

References in corpus (13)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Equality of Opportunity in Supervised Learning
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Fairness Without Demographics in Repeated Loss Minimization
• SelectiveNet: A Deep Neural Network with an Integrated Reject Option
• Noise or Signal: The Role of Image Backgrounds in Object Recognition
• Direct Uncertainty Prediction for Medical Second Opinions
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
• Consistent Estimators for Learning to Defer to an Expert
• xGEMs: Generating Examplars to Explain Black-Box Models