Model Selection's Disparate Impact in Real-World Deep Learning Applications

Abstract

Algorithmic fairness has emphasized the role of biased data in automated decision outcomes. Recently, there has been a shift in attention to sources of bias that implicate fairness in other stages in the ML pipeline. We contend that one source of such bias, human preferences in model selection, remains under-explored in terms of its role in disparate impact across demographic groups. Using a deep learning model trained on real-world medical imaging data, we verify our claim empirically and argue that choice of metric for model comparison, especially those that do not take variability into account, can significantly bias model selection outcomes.

arXiv commentScience and Engineering of Deep Learning Workshop, ICLR 2021

References in corpus (5)

• Practical Bayesian Optimization of Machine Learning Algorithms
• CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
• Sanity Checks for Saliency Maps
• Deep Reinforcement Learning that Matters
• Bringing the People Back In: Contesting Benchmark Machine Learning Datasets

Cited by in corpus (4)

• Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning
• Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
• Non-Determinism and the Lawlessness of Machine Learning Code
• Hyperparameter Optimization Is Deceiving Us, and How to Stop It