The Rich Get Richer: Disparate Impact of Semi-Supervised Learning
arXiv:2110.06282
Abstract
Semi-supervised learning (SSL) has demonstrated its potential to improve the model accuracy for a variety of learning tasks when the high-quality supervised data is severely limited. Although it is often established that the average accuracy for the entire population of data is improved, it is unclear how SSL fares with different sub-populations. Understanding the above question has substantial fairness implications when different sub-populations are defined by the demographic groups that we aim to treat fairly. In this paper, we reveal the disparate impacts of deploying SSL: the sub-population who has a higher baseline accuracy without using SSL (the "rich" one) tends to benefit more from SSL; while the sub-population who suffers from a low baseline accuracy (the "poor" one) might even observe a performance drop after adding the SSL module. We theoretically and empirically establish the above observation for a broad family of SSL algorithms, which either explicitly or implicitly use an auxiliary "pseudo-label". Experiments on a set of image and text classification tasks confirm our claims. We introduce a new metric, Benefit Ratio, and promote the evaluation of the fairness of SSL (Equalized Benefit Ratio). We further discuss how the disparate impact can be mitigated. We hope our paper will alarm the potential pitfall of using SSL and encourage a multifaceted evaluation of future SSL algorithms.
arXiv commentPublished as a conference paper at ICLR 2022. Revised constants Theorems 1,2, and Lemma 3 (consider the union bound). Add acknowledgments to Nautilus
References in corpus (9)
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Classification with Noisy Labels by Importance Reweighting
- Semi-Supervised Learning with Ladder Networks
- Semi-supervised Sequence Learning
- Weakly-Supervised Neural Text Classification
- What Do Compressed Deep Neural Networks Forget?
- Dash: Semi-Supervised Learning with Dynamic Thresholding
- Characterising Bias in Compressed Models