The Who in XAI: How AI Background Shapes Perceptions of AI Explanations

Abstract

Explainability of AI systems is critical for users to take informed actions. Understanding "who" opens the black-box of AI is just as important as opening it. We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. Quantitatively, we share user perceptions along five dimensions. Qualitatively, we describe how AI background can influence interpretations, elucidating the differences through lenses of appropriation and cognitive heuristics. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design. Carrying critical implications for the field of XAI, our findings showcase how AI generated explanations can have negative consequences despite best intentions and how that could lead to harmful manipulation of trust. We propose design interventions to mitigate them.

References in corpus (8)

• 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
• Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Problem Formulation and Fairness
• Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects
• Data Vision: Learning to See Through Algorithmic Abstraction
• Explainable AI for Robot Failures: Generating Explanations that Improve User Assistance in Fault Recovery
• Leveraging Rationales to Improve Human Task Performance

Cited by in corpus (6)

• Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
• Show Me the Work: Fact-Checkers' Requirements for Explainable Automated Fact-Checking
• The Centers and Margins of Modeling Humans in Well-being Technologies: A Decentering Approach
• People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI
• Leveraging Complementary AI Explanations to Mitigate Misunderstanding in XAI
• Better Together? The Role of Explanations in Supporting Novices in Individual and Collective Deliberations about AI