Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations

Abstract

Explainable AI (XAI) is widely viewed as a sine qua non for ever-expanding AI research. A better understanding of the needs of XAI users, as well as human-centered evaluations of explainable models are both a necessity and a challenge. In this paper, we explore how HCI and AI researchers conduct user studies in XAI applications based on a systematic literature review. After identifying and thoroughly analyzing 97core papers with human-based XAI evaluations over the past five years, we categorize them along the measured characteristics of explanatory methods, namely trust, understanding, usability, and human-AI collaboration performance. Our research shows that XAI is spreading more rapidly in certain application domains, such as recommender systems than in others, but that user evaluations are still rather sparse and incorporate hardly any insights from cognitive or social sciences. Based on a comprehensive discussion of best practices, i.e., common models, design choices, and measures in user studies, we propose practical guidelines on designing and conducting user studies for XAI researchers and practitioners. Lastly, this survey also highlights several open research directions, particularly linking psychological science and human-centered XAI.

References in corpus (30)

• Towards A Rigorous Science of Interpretable Machine Learning
• Methods for Interpreting and Understanding Deep Neural Networks
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• A Survey on the Explainability of Supervised Machine Learning
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
• 'It's Reducing a Human Being to a Percentage'; Perceptions of Justice in Algorithmic Decisions
• Towards Explainable Artificial Intelligence
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Expanding Explainability: Towards Social Transparency in AI systems
• A systematic review and taxonomy of explanations in decision support and recommender systems
• Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
• Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment
• Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models
• Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies
• Towards Relatable Explainable AI with the Perceptual Process
• Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions
• Leveraging Rationales to Improve Human Task Performance
• Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study
• Appropriate Fairness Perceptions? On the Effectiveness of Explanations in Enabling People to Assess the Fairness of Automated Decision Systems
• Beyond Individualized Recourse: Interpretable and Interactive Summaries of Actionable Recourses
• Keep Your Friends Close and Your Counterfactuals Closer: Improved Learning From Closest Rather Than Plausible Counterfactual Explanations in an Abstract Setting
• Let's Go to the Alien Zoo: Introducing an Experimental Framework to Study Usability of Counterfactual Explanations for Machine Learning
• Visual correspondence-based explanations improve AI robustness and human-AI team accuracy
• Use-Case-Grounded Simulations for Explanation Evaluation
• A Psychological Theory of Explainability
• User Trust on an Explainable AI-based Medical Diagnosis Support System
• Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset
• Quantifying Learnability and Describability of Visual Concepts Emerging in Representation Learning

Cited by in corpus (15)

• What Large Language Models Know and What People Think They Know
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing
• Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant
• How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps
• Towards Interpretability in Audio and Visual Affective Machine Learning: A Review
• Interpretability is in the eye of the beholder: Human versus artificial classification of image segments generated by humans versus XAI
• With Friends Like These, Who Needs Explanations? Evaluating User Understanding of Group Recommendations
• A Comprehensive Survey on Self-Interpretable Neural Networks
• Is My Data in Your AI? Membership Inference Test (MINT) applied to Face Biometrics
• Evaluating the Explainability of Attributes and Prototypes for a Medical Classification Model
• Class-Dependent Perturbation Effects in Evaluating Time Series Attributions
• Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks
• Explanation format does not matter; but explanations do -- An Eggsbert study on explaining Bayesian Optimisation tasks
• The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
• The Impact of Concept Explanations and Interventions on Human-Machine Collaboration