A Benchmark for Interpretability Methods in Deep Neural Networks

Abstract

We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks. Our results across several large-scale image classification datasets show that many popular interpretability methods produce estimates of feature importance that are not better than a random designation of feature importance. Only certain ensemble based approaches---VarGrad and SmoothGrad-Squared---outperform such a random assignment of importance. The manner of ensembling remains critical, we show that some approaches do no better then the underlying method but carry a far higher computational burden.

arXiv commentIn NeurIPS 2019

Cited by in corpus (106)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
• Post-hoc Interpretability for Neural NLP: A Survey
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• The impact of feature importance methods on the interpretation of defect classifiers
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• A Survey of Malware Detection Using Deep Learning
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
• What Do Compressed Deep Neural Networks Forget?
• Benchmarking Deep Learning Interpretability in Time Series Predictions
• Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?
• On the (In)fidelity and Sensitivity for Explanations
• A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• A Survey on Neural Network Interpretability
• Towards falsifiable interpretability research
• Towards Best Practice of Interpreting Deep Learning Models for EEG-based Brain Computer Interfaces
• What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors
• ferret: a Framework for Benchmarking Explainers on Transformers
• Explainable Natural Language Processing for Corporate Sustainability Analysis
• Dos and Don'ts of Machine Learning in Computer Security
• Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
• ZELDA: fitting Lyman-alpha line profiles using deep learning
• Selectivity considered harmful: evaluating the causal impact of class selectivity in DNNs
• FairCanary: Rapid Continuous Explainable Fairness
• Adversarial Infidelity Learning for Model Interpretation
• Why Should I Choose You? AutoXAI: A Framework for Selecting and Tuning eXplainable AI Solutions
• Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networks
• Generative causal explanations of black-box classifiers
• Fooling Partial Dependence via Data Poisoning
• One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images
• Human Attention in Fine-grained Classification
• On quantitative aspects of model interpretability
• Interpretable Machine Learning: Moving From Mythos to Diagnostics
• Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
• Spatio-Temporal Perturbations for Video Attribution
• Overinterpretation reveals image classification model pathologies
• IROF: a low resource evaluation metric for explanation methods
• Evaluating Input Perturbation Methods for Interpreting CNNs and Saliency Map Comparison
• Enhancing Deep Neural Network Saliency Visualizations with Gradual Extrapolation
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models
• Debiasing Concept-based Explanations with Causal Analysis
• Human-Centered Evaluation of XAI Methods
• A Baseline for Shapley Values in MLPs: from Missingness to Neutrality
• Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations
• Debugging Tests for Model Explanations
• A Game-Theoretic Taxonomy of Visual Concepts in DNNs
• EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Models
• Order in the Court: Explainable AI Methods Prone to Disagreement
• Achievements and Challenges in Explaining Deep Learning based Computer-Aided Diagnosis Systems
• Discriminating Spatial and Temporal Relevance in Deep Taylor Decompositions for Explainable Activity Recognition
• Improving Feature Attribution through Input-specific Network Pruning
• Evaluating Saliency Methods for Neural Language Models
• Data Representing Ground-Truth Explanations to Evaluate XAI Methods
• Explaining Motion Relevance for Activity Recognition in Video Deep Learning Models
• PassAI: explainable artificial intelligence algorithm for soccer pass analysis using multimodal information resources
• The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
• Improving Interpretability in Medical Imaging Diagnosis using Adversarial Training
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• Aggregating explanation methods for stable and robust explainability
• Analyzing Effects of Mixed Sample Data Augmentation on Model Interpretability
• Convolutional Neural Network Interpretability with General Pattern Theory
• Towards Robust Explanations for Deep Neural Networks
• An Empirical Study of Explainable AI Techniques on Deep Learning Models For Time Series Tasks
• Rethinking Positive Aggregation and Propagation of Gradients in Gradient-based Saliency Methods
• On the Explanatory Power of Decision Trees
• Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?
• GANMEX: One-vs-One Attributions Guided by GAN-based Counterfactual Explanation Baselines
• Neural Greedy Pursuit for Feature Selection
• Born Identity Network: Multi-way Counterfactual Map Generation to Explain a Classifier's Decision
• Identifying Critical Neurons in ANN Architectures using Mixed Integer Programming
• Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing
• A Novel Hybrid Feature Importance and Feature Interaction Detection Framework for Predictive Optimization in Industry 4.0 Applications
• Algorithm Fairness in AI for Medicine and Healthcare
• Factor Importance Ranking and Selection using Total Indices
• TREX: Tree-Ensemble Representer-Point Explanations
• Advancing Post Hoc Case Based Explanation with Feature Highlighting
• Evaluation of importance estimators in deep learning classifiers for Computed Tomography
• Why Attentions May Not Be Interpretable?
• On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluation
• Measuring and improving the quality of visual explanations
• Crowdsourcing Evaluation of Saliency-based XAI Methods
• Robust Semantic Interpretability: Revisiting Concept Activation Vectors
• DANCE: Enhancing saliency maps using decoys
• Scaling Symbolic Methods using Gradients for Neural Model Explanation
• Discriminative Attribution from Counterfactuals
• FW-Shapley: Real-time Estimation of Weighted Shapley Values
• Do Input Gradients Highlight Discriminative Features?
• On Spectral Properties of Gradient-based Explanation Methods
• Pattern-Guided Integrated Gradients
• The Generalizability of Explanations
• Sanity Simulations for Saliency Methods
• Longitudinal Distance: Towards Accountable Instance Attribution
• Towards Trustworthy Deception Detection: Benchmarking Model Robustness across Domains, Modalities, and Languages
• Explainability Requires Interactivity
• Learning Propagation Rules for Attribution Map Generation
• Coalitional Bayesian Autoencoders -- Towards explainable unsupervised deep learning
• Improving Attribution Methods by Learning Submodular Functions
• Inserting Information Bottlenecks for Attribution in Transformers
• Deep Synoptic Monte Carlo Planning in Reconnaissance Blind Chess
• Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation
• Explainability as statistical inference
• Towards Auditability for Fairness in Deep Learning
• Saliency strikes back: How filtering out high frequencies improves white-box explanations
• A Methodology for Exploring Deep Convolutional Features in Relation to Hand-Crafted Features with an Application to Music Audio Modeling
• Attribution Mask: Filtering Out Irrelevant Features By Recursively Focusing Attention on Inputs of DNNs