How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps
arXiv:2404.16042 · doi:10.1080/10447318.2024.2381929
Abstract
Saliency maps can explain how deep neural networks classify images. But are they actually useful for humans? The present systematic review of 68 user studies found that while saliency maps can enhance human performance, null effects or even costs are quite common. To investigate what modulates these effects, the empirical outcomes were organised along several factors related to the human tasks, AI performance, XAI methods, images to be classified, human participants and comparison conditions. In image-focused tasks, benefits were less common than in AI-focused tasks, but the effects depended on the specific cognitive requirements. Moreover, benefits were usually restricted to incorrect AI predictions in AI-focused tasks but to correct ones in image-focused tasks. XAI-related factors had surprisingly little impact. The evidence was limited for image- and human-related factors and the effects were highly dependent on the comparison conditions. These findings may support the design of future user studies.
References in corpus (20)
- Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
- Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
- Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations
- Explainable Artificial Intelligence (XAI) from a user perspective- A synthesis of prior literature and problematizing avenues for future research
- From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
- A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
- Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
- Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation
- Teaching Categories to Human Learners with Visual Explanations
- Embedding Deep Networks into Visual Explanations
- Trust and Reliance in XAI -- Distinguishing Between Attitudinal and Behavioral Measures
- Interpretability is in the eye of the beholder: Human versus artificial classification of image segments generated by humans versus XAI
- In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making
- The benefits and costs of explainable artificial intelligence in visual quality control: Evidence from fault detection performance and eye movements
- One Explanation is Not Enough: Structured Attention Graphs for Image Classification
- Minimalistic Explanations: Capturing the Essence of Decisions
- Modeling Reliance on XAI Indicating Its Purpose and Attention
- A psychophysics approach for quantitative comparison of interpretable computer vision models
- Perception Visualization: Seeing Through the Eyes of a DNN
- Responsibility: An Example-based Explainable AI approach via Training Process Inspection