It's only fair when I think it's fair: How Gender Bias Alignment Undermines Distributive Fairness in Human-AI Collaboration

Abstract

Human-AI collaboration is increasingly relevant in consequential areas where AI recommendations support human discretion. However, human-AI teams' effectiveness, capability, and fairness highly depend on human perceptions of AI. Positive fairness perceptions have been shown to foster trust and acceptance of AI recommendations. Yet, work on confirmation bias highlights that humans selectively adhere to AI recommendations that align with their expectations and beliefs -- despite not being necessarily correct or fair. This raises the question whether confirmation bias also transfers to the alignment of gender bias between human and AI decisions. In our study, we examine how gender bias alignment influences fairness perceptions and reliance. The results of a 2x2 between-subject study highlight the connection between gender bias alignment, fairness perceptions, and reliance, demonstrating that merely constructing a ``formally fair'' AI system is insufficient for optimal human-AI collaboration; ultimately, AI recommendations will likely be overridden if biases do not align.

References in corpus (13)

• Hybrid Intelligence
• 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
• Human-AI Interactions in Public Sector Decision-Making: "Automation Bias" and "Selective Adherence" to Algorithmic Advice
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• A Clarification of the Nuances in the Fairness Metrics Landscape
• Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
• This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology
• A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
• On the Quest for Effectiveness in Human Oversight: Interdisciplinary Perspectives
• Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making
• "There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making
• A Critical Survey on Fairness Benefits of Explainable AI
• On Prediction-Modelers and Decision-Makers: Why Fairness Requires More Than a Fair Prediction Model