Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data
arXiv:2103.03399
Abstract
Collecting more diverse and representative training data is often touted as a remedy for the disparate performance of machine learning predictors across subpopulations. However, a precise framework for understanding how dataset properties like diversity affect learning outcomes is largely lacking. By casting data collection as part of the learning process, we demonstrate that diverse representation in training data is key not only to increasing subgroup performances, but also to achieving population level objectives. Our analysis and experiments describe how dataset compositions influence performance and provide constructive results for using trends in existing data, alongside domain knowledge, to help guide intentional, objective-aware dataset design.
arXiv commentAccepted to ICML 2021; 31 pages,9 figures
References in corpus (12)
- A systematic study of the class imbalance problem in convolutional neural networks
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem
- Fairness Without Demographics in Repeated Loss Minimization
- No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World
- Data Shapley: Equitable Valuation of Data for Machine Learning
- Does Distributionally Robust Supervised Learning Give Robust Classifiers?
- A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle
- InclusiveFaceNet: Improving Face Attribute Detection with Race and Gender Diversity
- Are All Training Examples Created Equal? An Empirical Study
- No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems
- Who's responsible? Jointly quantifying the contribution of the learning algorithm and training data