Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems

Abstract

Explainable artificially intelligent (XAI) systems form part of sociotechnical systems, e.g., human+AI teams tasked with making decisions. Yet, current XAI systems are rarely evaluated by measuring the performance of human+AI teams on actual decision-making tasks. We conducted two online experiments and one in-person think-aloud study to evaluate two currently common techniques for evaluating XAI systems: (1) using proxy, artificial tasks such as how well humans predict the AI's decision from the given explanations, and (2) using subjective measures of trust and preference as predictors of actual performance. The results of our experiments demonstrate that evaluations with proxy tasks did not predict the results of the evaluations with the actual decision-making tasks. Further, the subjective measures on evaluations with actual decision-making tasks did not predict the objective performance on those same tasks. Our results suggest that by employing misleading evaluation methods, our field may be inadvertently slowing its progress toward developing human+AI teams that can reliably perform better than humans or AIs alone.

References in corpus (1)

• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection

Cited by in corpus (35)

• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
• Expanding Explainability: Towards Social Transparency in AI systems
• When combinations of humans and AI are useful: A systematic review and meta-analysis
• Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations
• Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens
• Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
• Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction
• The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
• Investigating and Designing for Trust in AI-powered Code Generation Tools
• The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
• Twenty-Four Years of Empirical Research on Trust in AI: A Bibliometric Review of Trends, Overlooked Issues, and Future Directions
• Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
• Participation in the age of foundation models
• An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability
• Outcome-Explorer: A Causality Guided Interactive Visual Interface for Interpretable Algorithmic Decision Making
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure
• GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information
• How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps
• White Box Methods for Explanations of Convolutional Neural Networks in Image Classification Tasks
• AI, Help Me Think$\unicode{x2014}$but for Myself: Assisting People in Complex Decision-Making by Providing Different Kinds of Cognitive Support
• Calibrated Explanations: with Uncertainty Information and Counterfactuals
• Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models
• Towards a Comprehensive Human-Centred Evaluation Framework for Explainable AI
• Information That Matters: Exploring Information Needs of People Affected by Algorithmic Decisions
• LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
• The Impact of Explanations on Fairness in Human-AI Decision-Making: Protected vs Proxy Features
• With Friends Like These, Who Needs Explanations? Evaluating User Understanding of Group Recommendations
• Calibrated Explanations for Regression
• People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI
• Evaluating Model Explanations without Ground Truth
• Give Me a Choice: The Consequences of Restricting Choices Through AI-Support for Perceived Autonomy, Motivational Variables, and Decision Performance
• Why is "Problems" Predictive of Positive Sentiment? A Case Study of Explaining Unintuitive Features in Sentiment Classification
• On the Definition of Appropriate Trust and the Tools that Come with it
• Beyond Satisfaction: From Placebic to Actionable Explanations For Enhanced Understandability