"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction

Abstract

Despite the proliferation of explainable AI (XAI) methods, little is understood about end-users' explainability needs and behaviors around XAI explanations. To address this gap and contribute to understanding how explainability can support human-AI interaction, we conducted a mixed-methods study with 20 end-users of a real-world AI application, the Merlin bird identification app, and inquired about their XAI needs, uses, and perceptions. We found that participants desire practically useful information that can improve their collaboration with the AI, more so than technical system details. Relatedly, participants intended to use XAI explanations for various purposes beyond understanding the AI's outputs: calibrating trust, improving their task skills, changing their behavior to supply better inputs to the AI, and giving constructive feedback to developers. Finally, among existing XAI approaches, participants preferred part-based explanations that resemble human reasoning and explanations. We discuss the implications of our findings and provide recommendations for future XAI design.

CHI 2023

References in corpus (16)

• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research
• Expanding Explainability: Towards Social Transparency in AI systems
• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
• Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
• StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement
• Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
• Towards Relatable Explainable AI with the Perceptual Process
• Survey of XAI in digital pathology
• Trust and Reliance in XAI -- Distinguishing Between Attitudinal and Behavioral Measures
• Are we measuring trust correctly in explainability, interpretability, and transparency research?
• Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
• On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods
• ELUDE: Generating interpretable explanations via a decomposition into labelled and unlabelled features
• Overlooked factors in concept-based explanations: Dataset choice, concept learnability, and human capability

Cited by in corpus (18)

• Digital Deception: Generative Artificial Intelligence in Social Engineering and Phishing
• The Impact of Imperfect XAI on Human-AI Decision-Making
• Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
• Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
• EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations
• Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
• Task Supportive and Personalized Human-Large Language Model Interaction: A User Study
• Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning
• Concept-based Explainable Artificial Intelligence: A Survey
• Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software
• Interrogating AI: Characterizing Emergent Playful Interactions with ChatGPT
• Identifying Explanation Needs of End-users: Applying and Extending the XAI Question Bank
• People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI
• Design Considerations for Human Oversight of AI: Insights from Co-Design Workshops and Work Design Theory
• GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design
• What Lies Beneath? Exploring the Impact of Underlying AI Model Updates in AI-Infused Systems
• Black-box Attacks on Image Activity Prediction and its Natural Language Explanations
• Integrating measures of replicability into scholarly search: Challenges and opportunities