Bias In, Bias Out? Evaluating the Folk Wisdom

Abstract

We evaluate the folk wisdom that algorithmic decision rules trained on data produced by biased human decision-makers necessarily reflect this bias. We consider a setting where training labels are only generated if a biased decision-maker takes a particular action, and so "biased" training data arise due to discriminatory selection into the training data. In our baseline model, the more biased the decision-maker is against a group, the more the algorithmic decision rule favors that group. We refer to this phenomenon as "bias reversal." We then clarify the conditions that give rise to bias reversal. Whether a prediction algorithm reverses or inherits bias depends critically on how the decision-maker affects the training data as well as the label used in training. We illustrate our main theoretical results in a simulation study applied to the New York City Stop, Question and Frisk dataset.

References in corpus (5)

• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• Avoiding Discrimination through Causal Reasoning
• Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination
• A Decision Theoretic Approach to Targeted Advertising
• Use Privacy in Data-Driven Systems: Theory and Experiments with Machine Learnt Programs

Cited by in corpus (9)

• The Measure and Mismeasure of Fairness
• The Disparate Effects of Strategic Manipulation
• FairSight: Visual Analytics for Fairness in Decision Making
• Bias in Machine Learning -- What is it Good for?
• A survey on datasets for fairness-aware machine learning
• Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education?
• Modeling Techniques for Machine Learning Fairness: A Survey
• Dynamic Modeling and Equilibria in Fair Decision Making
• Hidden Technical Debts for Fair Machine Learning in Financial Services