Questioning the AI: Informing Design Practices for Explainable AI User Experiences

Abstract

A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many recognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for understanding AI remains an open question. By interviewing 20 UX and design practitioners working on various AI products, we seek to identify gaps between the current XAI algorithmic work and practices to create explainable AI products. To do so, we develop an algorithm-informed XAI question bank in which user needs for explainability are represented as prototypical questions users might ask about the AI, and use it as a study probe. Our work contributes insights into the design space of XAI, informs efforts to support design practices in this space, and identifies opportunities for future XAI work. We also provide an extended XAI question bank and discuss how it can be used for creating user-centered XAI.

arXiv commentPublishedACM CHI Conference on Human Factors in Computing Systems (CHI 2020). Updated XAI Question Bank in September 2021

References in corpus (5)

• Improving fairness in machine learning systems: What do industry practitioners need?
• 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Prototype selection for interpretable classification
• Interpretable Predictions of Tree-based Ensembles via Actionable Feature Tweaking

Cited by in corpus (78)

• Expanding Explainability: Towards Social Transparency in AI systems
• "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
• Design Principles for Generative AI Applications
• "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
• Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquiry
• Explainable Matrix -- Visualization for Global and Local Interpretability of Random Forest Classification Ensembles
• Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems
• Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
• PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
• Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook
• Perfection Not Required? Human-AI Partnerships in Code Translation
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
• "HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media
• Secure and Trustworthy Artificial Intelligence-Extended Reality (AI-XR) for Metaverses
• Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosis
• Trustworthy AI: Securing Sensitive Data in Large Language Models
• Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
• "It Felt Like Having a Second Mind": Investigating Human-AI Co-creativity in Prewriting with Large Language Models
• Investigating and Designing for Trust in AI-powered Code Generation Tools
• Towards Unifying Feature Attribution and Counterfactual Explanations: Different Means to the Same End
• XAIR: A Framework of Explainable AI in Augmented Reality
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Better Together? An Evaluation of AI-Supported Code Translation
• Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
• Towards Relatable Explainable AI with the Perceptual Process
• AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates
• Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work
• Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
• Certification Labels for Trustworthy AI: Insights From an Empirical Mixed-Method Study
• Directive Explanations for Monitoring the Risk of Diabetes Onset: Introducing Directive Data-Centric Explanations and Combinations to Support What-If Explorations
• Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
• Seamful XAI: Operationalizing Seamful Design in Explainable AI
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India
• EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations
• Effects of Multimodal Explanations for Autonomous Driving on Driving Performance, Cognitive Load, Expertise, Confidence, and Trust
• Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
• Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
• AutoML in The Wild: Obstacles, Workarounds, and Expectations
• Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences
• Co-designing Large Language Model Tools for Project-Based Learning with K12 Educators
• Exploring Variational Auto-Encoder Architectures, Configurations, and Datasets for Generative Music Explainable AI
• AI-Assisted Causal Pathway Diagram for Human-Centered Design
• Explanation Strategies as an Empirical-Analytical Lens for Socio-Technical Contextualization of Machine Learning Interpretability
• How Can We Develop Explainable Systems? Insights from a Literature Review and an Interview Study
• Can Requirements Engineering Support Explainable Artificial Intelligence? Towards a User-Centric Approach for Explainability Requirements
• Explainable AI Reloaded: Challenging the XAI Status Quo in the Era of Large Language Models
• Explaining Any ML Model? -- On Goals and Capabilities of XAI
• The AI-DEC: A Card-based Design Method for User-centered AI Explanations
• How Learner Control and Explainable Learning Analytics on Skill Mastery Shape Student Desires to Finish and Avoid Loss in Tutored Practice
• From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent
• XAIxArts Manifesto: Explainable AI for the Arts
• Towards Balancing Preference and Performance through Adaptive Personalized Explainability
• Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software
• Explainable AI-Based Interface System for Weather Forecasting Model
• Identifying Explanation Needs of End-users: Applying and Extending the XAI Question Bank
• Study on the Helpfulness of Explainable Artificial Intelligence
• AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients
• People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI
• From Heatmaps to Structural Explanations of Image Classifiers
• DeepSeer: Interactive RNN Explanation and Debugging via State Abstraction
• "Why do we do this?": Moral Stress and the Affective Experience of Ethics in Practice
• Negotiating the Shared Agency between Humans & AI in the Recommender System
• A three-Level Framework for LLM-Enhanced eXplainable AI: From technical explanations to natural language
• Generating Context-Aware Contrastive Explanations in Rule-based Systems
• Design Considerations for Human Oversight of AI: Insights from Co-Design Workshops and Work Design Theory
• Dissecting users' needs for search result explanations
• Self-Explanation in Social AI Agents
• Better Together? The Role of Explanations in Supporting Novices in Individual and Collective Deliberations about AI
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems
• VibE: A Visual Analytics Workflow for Semantic Error Analysis of CVML Models at Subgroup Level
• Investigating the Capabilities and Limitations of Machine Learning for Identifying Bias in English Language Data with Information and Heritage Professionals
• Co-Disclosing the Computer: LLM-Mediated Computing through Reflective Conversation
• A Human-Centered Privacy Approach (HCP) to AI
• Designing Visual Explanations and Learner Controls to Engage Adolescents in AI-Supported Exercise Selection
• To LLM, or Not to LLM: How Designers and Developers Navigate LLMs as Tools or Teammates
• Visual Text Mining with Progressive Taxonomy Construction for Environmental Studies
• Explainability Assistant: A Conversational XAI Interface for Interpreting Energy Consumption Models