Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference

Abstract

We investigate the problem of reliably assessing group fairness when labeled examples are few but unlabeled examples are plentiful. We propose a general Bayesian framework that can augment labeled data with unlabeled data to produce more accurate and lower-variance estimates compared to methods based on labeled data alone. Our approach estimates calibrated scores for unlabeled examples in each group using a hierarchical latent variable model conditioned on labeled examples. This in turn allows for inference of posterior distributions with associated notions of uncertainty for a variety of group fairness metrics. We demonstrate that our approach leads to significant and consistent reductions in estimation error across multiple well-known fairness datasets, sensitive attributes, and predictive models. The results show the benefits of using both unlabeled data and Bayesian inference in terms of assessing whether a prediction model is fair or not.

arXiv comment27 pages

References in corpus (8)

• Equality of Opportunity in Supervised Learning
• On Calibration of Modern Neural Networks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• The Measure and Mismeasure of Fairness
• Calibrating general posterior credible regions
• Leveraging Labeled and Unlabeled Data for Consistent Fair Binary Classification
• Confidence Intervals for Testing Disparate Impact in Fair Learning
• Bayesian Modeling of Intersectional Fairness: The Variance of Bias

Cited by in corpus (5)

• Toward Operationalizing Pipeline-aware ML Fairness: A Research Agenda for Developing Practical Guidelines and Tools
• Fairness Hacking: The Malicious Practice of Shrouding Unfairness in Algorithms
• Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance
• (Un)certainty of (Un)fairness: Preference-Based Selection of Certainly Fair Decision-Makers
• Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations