Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI

Abstract

The rise of deep learning in today's applications entailed an increasing need in explaining the model's decisions beyond prediction performances in order to foster trust and accountability. Recently, the field of explainable AI (XAI) has developed methods that provide such explanations for already trained neural networks. In computer vision tasks such explanations, termed heatmaps, visualize the contributions of individual pixels to the prediction. So far XAI methods along with their heatmaps were mainly validated qualitatively via human-based assessment, or evaluated through auxiliary proxy tasks such as pixel perturbation, weak object localization or randomization tests. Due to the lack of an objective and commonly accepted quality measure for heatmaps, it was debatable which XAI method performs best and whether explanations can be trusted at all. In the present work, we tackle the problem by proposing a ground truth based evaluation framework for XAI methods based on the CLEVR visual question answering task. Our framework provides a (1) selective, (2) controlled and (3) realistic testbed for the evaluation of neural network explanations. We compare ten different explanation methods, resulting in new insights about the quality and properties of XAI methods, sometimes contradicting with conclusions from previous comparative studies. The CLEVR-XAI dataset and the benchmarking code can be found at https://github.com/ahmedmagdiosman/clevr-xai.

37 pages, 9 tables, 2 figures (plus appendix 14 pages)

References in corpus (10)

• Methods for Interpreting and Understanding Deep Neural Networks
• Striving for Simplicity: The All Convolutional Net
• Shortcut Learning in Deep Neural Networks
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• SmoothGrad: removing noise by adding noise
• A simple neural network module for relational reasoning
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments

Cited by in corpus (25)

• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
• Adversarial attacks and defenses in explainable artificial intelligence: A survey
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathology
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing
• An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability
• Let's Go to the Alien Zoo: Introducing an Experimental Framework to Study Usability of Counterfactual Explanations for Machine Learning
• Evaluating the Effectiveness of XAI Techniques for Encoder-Based Language Models
• CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
• Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification
• Interpretability is in the eye of the beholder: Human versus artificial classification of image segments generated by humans versus XAI
• Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations
• The benefits and costs of explainable artificial intelligence in visual quality control: Evidence from fault detection performance and eye movements
• Saliency-Bench: A Comprehensive Benchmark for Evaluating Visual Explanations
• What Does an Audio Deepfake Detector Focus on? A Study in the Time Domain
• On The Coherence of Quantitative Evaluation of Visual Explanations
• XAI-CF -- Examining the Role of Explainable Artificial Intelligence in Cyber Forensics
• XAI-Units: Benchmarking Explainability Methods with Unit Tests
• PCIM: Learning Pixel Attributions via Pixel-wise Channel Isolation Mixing in High Content Imaging
• Towards the Characterization of Representations Learned via Capsule-based Network Architectures
• GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations
• Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity
• Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation
• Metric-Guided Synthesis of Class Activation Mapping