Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models
arXiv:2404.07725 · doi:10.1145/3613904.3642934
Abstract
The field of eXplainable artificial intelligence (XAI) has produced a plethora of methods (e.g., saliency-maps) to gain insight into artificial intelligence (AI) models, and has exploded with the rise of deep learning (DL). However, human-participant studies question the efficacy of these methods, particularly when the AI output is wrong. In this study, we collected and analyzed 156 human-generated text and saliency-based explanations collected in a question-answering task (N=40) and compared them empirically to state-of-the-art XAI explanations (integrated gradients, conservative LRP, and ChatGPT) in a human-participant study (N=136). Our findings show that participants found human saliency maps to be more helpful in explaining AI answers than machine saliency maps, but performance negatively correlated with trust in the AI model and explanations. This finding hints at the dilemma of AI errors in explanation, where helpful explanations can lead to lower task performance when they support wrong AI predictions.
References in corpus (8)
- Towards A Rigorous Science of Interpretable Machine Learning
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
- Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
- One Explanation Does Not Fit All: The Promise of Interactive Explanations for Machine Learning Transparency
- A Meta-Analysis of the Utility of Explainable Artificial Intelligence in Human-AI Decision-Making
- Explainability Pitfalls: Beyond Dark Patterns in Explainable AI
- In Search of Verifiability: Explanations Rarely Enable Complementary Performance in AI-Advised Decision Making