Auditing ML Models for Individual Bias and Unfairness

Abstract

We consider the task of auditing ML models for individual bias/unfairness. We formalize the task in an optimization problem and develop a suite of inferential tools for the optimal value. Our tools permit us to obtain asymptotic confidence intervals and hypothesis tests that cover the target/control the Type I error rate exactly. To demonstrate the utility of our tools, we use them to reveal the gender and racial biases in Northpointe's COMPAS recidivism prediction instrument.

arXiv commentIn Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) 2020

References in corpus (4)

• Certifying Some Distributional Robustness with Principled Adversarial Training
• Minimax Statistical Learning with Wasserstein Distances
• Training individually fair ML models with Sensitive Subspace Robustness
• Confidence Regions in Wasserstein Distributionally Robust Estimation

Cited by in corpus (4)

• SenSeI: Sensitive Set Invariance for Enforcing Individual Fairness
• Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness
• Everything is Relative: Understanding Fairness with Optimal Transport
• Fair Representation Learning using Interpolation Enabled Disentanglement