Seamful XAI: Operationalizing Seamful Design in Explainable AI

Abstract

Mistakes in AI systems are inevitable, arising from both technical limitations and sociotechnical gaps. While black-boxing AI systems can make the user experience seamless, hiding the seams risks disempowering users to mitigate fallouts from AI mistakes. Instead of hiding these AI imperfections, can we leverage them to help the user? While Explainable AI (XAI) has predominantly tackled algorithmic opaqueness, we propose that seamful design can foster AI explainability by revealing and leveraging sociotechnical and infrastructural mismatches. We introduce the concept of Seamful XAI by (1) conceptually transferring "seams" to the AI context and (2) developing a design process that helps stakeholders anticipate and design with seams. We explore this process with 43 AI practitioners and real end-users, using a scenario-based co-design activity informed by real-world use cases. We found that the Seamful XAI design process helped users foresee AI harms, identify underlying reasons (seams), locate them in the AI's lifecycle, learn how to leverage seamful information to improve XAI and user agency. We share empirical insights, implications, and reflections on how this process can help practitioners anticipate and craft seams in AI, how seamfulness can improve explainability, empower end-users, and facilitate Responsible AI.

References in corpus (16)

• Model Cards for Model Reporting
• Improving fairness in machine learning systems: What do industry practitioners need?
• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• Expanding Explainability: Towards Social Transparency in AI systems
• Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Problem Formulation and Fairness
• Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
• Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects
• Designing for Human Rights in AI
• Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens
• CheXplain: Enabling Physicians to Explore and UnderstandData-Driven, AI-Enabled Medical Imaging Analysis
• Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
• Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
• Undoing Seamlessness: Exploring Seams for Critical Visualization

Cited by in corpus (3)

• Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
• Agency Among Agents: Designing with Hypertextual Friction in the Algorithmic Web
• Give Me a Choice: The Consequences of Restricting Choices Through AI-Support for Perceived Autonomy, Motivational Variables, and Decision Performance