Explanations, Fairness, and Appropriate Reliance in Human-AI Decision-Making

Abstract

In this work, we study the effects of feature-based explanations on distributive fairness of AI-assisted decisions, specifically focusing on the task of predicting occupations from short textual bios. We also investigate how any effects are mediated by humans' fairness perceptions and their reliance on AI recommendations. Our findings show that explanations influence fairness perceptions, which, in turn, relate to humans' tendency to adhere to AI recommendations. However, we see that such explanations do not enable humans to discern correct and incorrect AI recommendations. Instead, we show that they may affect reliance irrespective of the correctness of AI recommendations. Depending on which features an explanation highlights, this can foster or hinder distributive fairness: when explanations highlight features that are task-irrelevant and evidently associated with the sensitive attribute, this prompts overrides that counter AI recommendations that align with gender stereotypes. Meanwhile, if explanations appear task-relevant, this induces reliance behavior that reinforces stereotype-aligned errors. These results imply that feature-based explanations are not a reliable mechanism to improve distributive fairness.

ACM CHI Conference on Human Factors in Computing Systems (CHI '24)

References in corpus (16)

• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
• Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment
• What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research
• Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations
• Auditing for Discrimination in Algorithms Delivering Job Ads
• A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
• Tackling Algorithmic Disability Discrimination in the Hiring Process: An Ethical, Legal and Technical Analysis
• Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
• "There Is Not Enough Information": On the Effects of Explanations on Perceptions of Informational Fairness and Trustworthiness in Automated Decision-Making
• A Critical Survey on Fairness Benefits of Explainable AI
• Making Fair ML Software using Trustworthy Explanation
• Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness

Cited by in corpus (5)

• The Impact of Imperfect XAI on Human-AI Decision-Making
• User Experience with LLM-powered Conversational Recommendation Systems: A Case of Music Recommendation
• It's only fair when I think it's fair: How Gender Bias Alignment Undermines Distributive Fairness in Human-AI Collaboration
• Give Me a Choice: The Consequences of Restricting Choices Through AI-Support for Perceived Autonomy, Motivational Variables, and Decision Performance
• Perils of Label Indeterminacy: A Case Study on Prediction of Neurological Recovery After Cardiac Arrest