Sanity Checks for Saliency Maps

Abstract

Saliency methods have emerged as a popular tool to highlight features in an input deemed relevant for the prediction of a learned model. Several saliency methods have been proposed, often guided by visual appeal on image data. In this work, we propose an actionable methodology to evaluate what kinds of explanations a given method can and cannot provide. We find that reliance, solely, on visual assessment can be misleading. Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. Consequently, methods that fail the proposed tests are inadequate for tasks that are sensitive to either data or model, such as, finding outliers in the data, explaining the relationship between inputs and outputs that the model learned, and debugging the model. We interpret our findings through an analogy with edge detection in images, a technique that requires neither training data nor model. Theory in the case of a linear model and a single-layer convolutional neural network supports our experimental findings.

Updating Guided Backprop experiments due to bug. The results and conclusions remain the same

Cited by in corpus (244)

• Interpretable machine learning: definitions, methods, and applications
• Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
• GNNExplainer: Generating Explanations for Graph Neural Networks
• Captum: A unified and generic model interpretability library for PyTorch
• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Analyzing Federated Learning through an Adversarial Lens
• Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection
• Concept Whitening for Interpretable Image Recognition
• What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
• Towards Out-Of-Distribution Generalization: A Survey
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Evaluating explainable artificial intelligence methods for multi-label deep learning classification tasks in remote sensing
• XCM: An Explainable Convolutional Neural Network for Multivariate Time Series Classification
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• Acquisition of Chess Knowledge in AlphaZero
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• Learning and Evaluating Representations for Deep One-class Classification
• Explainable Artificial Intelligence: a Systematic Review
• Benchmarking Deep Learning Interpretability in Time Series Predictions
• Benchmarking Attribution Methods with Relative Feature Importance
• Towards Automatic Concept-based Explanations
• Deep neural network or dermatologist?
• ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery
• Context-encoding Variational Autoencoder for Unsupervised Anomaly Detection
• "Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations
• Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning
• Adversarial Robustness as a Prior for Learned Representations
• Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
• On the (In)fidelity and Sensitivity for Explanations
• Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps
• Restricting the Flow: Information Bottlenecks for Attribution
• Practical Solutions for Machine Learning Safety in Autonomous Vehicles
• Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation
• A Human-Grounded Evaluation Benchmark for Local Explanations of Machine Learning
• On Interpretability of Artificial Neural Networks: A Survey
• Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey
• When Explanations Lie: Why Many Modified BP Attributions Fail
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• Regularizing Black-box Models for Improved Interpretability
• Counterfactual Visual Explanations
• NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks
• Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
• Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
• Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks
• Interpreting Black Box Models via Hypothesis Testing
• Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
• A Survey on Neural Network Interpretability
• White Box Methods for Explanations of Convolutional Neural Networks in Image Classification Tasks
• The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
• What Do You See? Evaluation of Explainable Artificial Intelligence (XAI) Interpretability through Neural Backdoors
• Smoothed Geometry for Robust Attribution
• Bridging Adversarial Robustness and Gradient Interpretability
• Explaining Time Series Predictions with Dynamic Masks
• Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
• What can AI do for me: Evaluating Machine Learning Interpretations in Cooperative Play
• Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency
• Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
• Now You See Me (CME): Concept-based Model Extraction
• Explaining Deep Neural Networks
• Saliency Methods for Explaining Adversarial Attacks
• Model extraction from counterfactual explanations
• Deep Stable Learning for Out-Of-Distribution Generalization
• Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays
• Fast Hierarchical Games for Image Explanations
• Evaluation of Similarity-based Explanations
• Interpretable Deep Learning under Fire
• Adversarial Infidelity Learning for Model Interpretation
• Interpreting Super-Resolution Networks with Local Attribution Maps
• Deep Learning in Protein Structural Modeling and Design
• Certifiably Robust Interpretation in Deep Learning
• Mutual Information for Explainable Deep Learning of Multiscale Systems
• On Baselines for Local Feature Attributions
• Evaluations and Methods for Explanation through Robustness Analysis
• IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography
• Predicting Model Failure using Saliency Maps in Autonomous Driving Systems
• Finding Experts in Transformer Models
• Weakly-Supervised Action Localization and Action Recognition using Global-Local Attention of 3D CNN
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
• One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images
• A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case
• Visual Explanation for Deep Metric Learning
• Directional convergence and alignment in deep learning
• Use HiResCAM instead of Grad-CAM for faithful explanations of convolutional neural networks
• Quantitative Evaluations on Saliency Methods: An Experimental Study
• Fooling Partial Dependence via Data Poisoning
• Best of both worlds: local and global explanations with human-understandable concepts
• Graph Neural Network for Interpreting Task-fMRI Biomarkers
• 3DB: A Framework for Debugging Computer Vision Models
• GCExplainer: Human-in-the-Loop Concept-based Explanations for Graph Neural Networks
• Explanation by Progressive Exaggeration
• Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation
• Causal Interpretability for Machine Learning -- Problems, Methods and Evaluation
• Personalized explanation in machine learning: A conceptualization
• Revisiting Sanity Checks for Saliency Maps
• Interpretable Machine Learning: Moving From Mythos to Diagnostics
• A Holistic Approach to Interpretability in Financial Lending: Models, Visualizations, and Summary-Explanations
• Question-Conditioned Counterfactual Image Generation for VQA
• Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
• Rethinking and Improving the Robustness of Image Style Transfer
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• Sparse Oblique Decision Trees: A Tool to Understand and Manipulate Neural Net Features
• Certification of embedded systems based on Machine Learning: A survey
• Trustworthy Convolutional Neural Networks: A Gradient Penalized-based Approach
• Model Selection's Disparate Impact in Real-World Deep Learning Applications
• Interpreting Deep Neural Networks Through Variable Importance
• IROF: a low resource evaluation metric for explanation methods
• Overinterpretation reveals image classification model pathologies
• Computationally Efficient Feature Significance and Importance for Machine Learning Models
• Incorporating Priors with Feature Attribution on Text Classification
• Investigating Bias in Image Classification using Model Explanations
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models
• Attribution-driven Causal Analysis for Detection of Adversarial Examples
• A simple defense against adversarial attacks on heatmap explanations
• Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations
• Reliable and Trustworthy Machine Learning for Health Using Dataset Shift Detection
• Semantics for Global and Local Interpretation of Deep Neural Networks
• Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
• XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanations
• Towards a Unified Evaluation of Explanation Methods without Ground Truth
• Debugging Tests for Model Explanations
• Achievements and Challenges in Explaining Deep Learning based Computer-Aided Diagnosis Systems
• CAMERAS: Enhanced Resolution And Sanity preserving Class Activation Mapping for image saliency
• Information-Theoretic Visual Explanation for Black-Box Classifiers
• Interpreting Adversarial Examples with Attributes
• Improving Feature Attribution through Input-specific Network Pruning
• A Survey on the Robustness of Feature Importance and Counterfactual Explanations
• Hide-and-Seek: A Template for Explainable AI
• Interpretable Graph Capsule Networks for Object Recognition
• Robust Models Are More Interpretable Because Attributions Look Normal
• Evaluating Saliency Methods for Neural Language Models
• Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models
• Robust and Stable Black Box Explanations
• Data Representing Ground-Truth Explanations to Evaluate XAI Methods
• Interpreting Interpretations: Organizing Attribution Methods by Criteria
• Preserve, Promote, or Attack? GNN Explanation via Topology Perturbation
• Aggregating explanation methods for stable and robust explainability
• Towards the Unification and Robustness of Perturbation and Gradient Based Explanations
• Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
• Improving 3D convolutional neural network comprehensibility via interactive visualization of relevance maps: Evaluation in Alzheimer's disease
• A Simple Saliency Method That Passes the Sanity Checks
• Optimising for Interpretability: Convolutional Dynamic Alignment Networks
• Grounding Representation Similarity with Statistical Testing
• An Empirical Study of Explainable AI Techniques on Deep Learning Models For Time Series Tasks
• Gradient-based Analysis of NLP Models is Manipulable
• TX-Ray: Quantifying and Explaining Model-Knowledge Transfer in (Un-)Supervised NLP
• Global Saliency: Aggregating Saliency Maps to Assess Dataset Artefact Bias
• Feature Attributions and Counterfactual Explanations Can Be Manipulated
• Rethinking Positive Aggregation and Propagation of Gradients in Gradient-based Saliency Methods
• Towards Robust Explanations for Deep Neural Networks
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
• Adaptive wavelet distillation from neural networks through interpretations
• Are Gradient-based Saliency Maps Useful in Deep Reinforcement Learning?
• Knowledge-based XAI through CBR: There is more to explanations than models can tell
• Understanding Learned Reward Functions
• TSInsight: A local-global attribution framework for interpretability in time-series data
• Challenges for cognitive decoding using deep learning methods
• How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels
• Quantifying Learnability and Describability of Visual Concepts Emerging in Representation Learning
• Finding and Fixing Spurious Patterns with Explanations
• Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling
• GANMEX: One-vs-One Attributions Guided by GAN-based Counterfactual Explanation Baselines
• Towards Musically Meaningful Explanations Using Source Separation
• Leveraging Sparse Linear Layers for Debuggable Deep Networks
• Manifold Coordinates with Physical Meaning
• Learning from learning machines: a new generation of AI technology to meet the needs of science
• Towards Rigorous Interpretations: a Formalisation of Feature Attribution
• Graph Neural Networks Including Sparse Interpretability
• Connecting Interpretability and Robustness in Decision Trees through Separation
• Quantitative and Qualitative Evaluation of Explainable Deep Learning Methods for Ophthalmic Diagnosis
• Towards a Resilient Machine Learning Classifier -- a Case Study of Ransomware Detection
• Crowdsourcing Evaluation of Saliency-based XAI Methods
• Feature Importance in a Deep Learning Climate Emulator
• On the Veracity of Local, Model-agnostic Explanations in Audio Classification: Targeted Investigations with Adversarial Examples
• Evaluation of Saliency-based Explainability Method
• EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case
• Towards Grad-CAM Based Explainability in a Legal Text Processing Pipeline
• iGOS++: Integrated Gradient Optimized Saliency by Bilateral Perturbations
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• A Bandit Model for Human-Machine Decision Making with Private Information and Opacity
• Robust Semantic Interpretability: Revisiting Concept Activation Vectors
• Socioeconomic correlations of urban patterns inferred from aerial images: interpreting activation maps of Convolutional Neural Networks
• Measuring and improving the quality of visual explanations
• Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI
• An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness
• Attribution Methods Reveal Flaws in Fingerprint-Based Virtual Screening
• Switched linear projections for neural network interpretability
• Explaining Image Classifiers using Statistical Fault Localization
• Visualization approach to assess the robustness of neural networks for medical image classification
• What Do Adversarially Robust Models Look At?
• Regularizing Black-box Models for Improved Interpretability (HILL 2019 Version)
• Gradient Frequency Modulation for Visually Explaining Video Understanding Models
• Discriminative Attribution from Counterfactuals
• Assessing the Reliability of Visual Explanations of Deep Models with Adversarial Perturbations
• Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated
• Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems
• DANCE: Enhancing saliency maps using decoys
• Deep Learning Based Decision Support for Medicine -- A Case Study on Skin Cancer Diagnosis
• Pattern-Guided Integrated Gradients
• Do Input Gradients Highlight Discriminative Features?
• A Tour of Convolutional Networks Guided by Linear Interpreters
• GAM: Explainable Visual Similarity and Classification via Gradient Activation Maps
• Introspective Learning by Distilling Knowledge from Online Self-explanation
• Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments
• Visualizing Color-wise Saliency of Black-Box Image Classification Models
• Explaining Deep Learning Representations by Tracing the Training Process
• MEME: Generating RNN Model Explanations via Model Extraction
• Semantic Network Interpretation
• Reducing Unintended Bias of ML Models on Tabular and Textual Data
• Interpretable ECG classification via a query-based latent space traversal (qLST)
• Learning Invariants through Soft Unification
• Defense Against Explanation Manipulation
• Do not explain without context: addressing the blind spot of model explanations
• A Methodology for Exploring Deep Convolutional Features in Relation to Hand-Crafted Features with an Application to Music Audio Modeling
• Characterizing the risk of fairwashing
• Deep Synoptic Monte Carlo Planning in Reconnaissance Blind Chess
• Longitudinal Distance: Towards Accountable Instance Attribution
• Human-Understandable Decision Making for Visual Recognition
• Generalizable and Explainable Dialogue Generation via Explicit Action Learning
• CACTUS: Detecting and Resolving Conflicts in Objective Functions
• Discovering Invariances in Healthcare Neural Networks
• Learning Propagation Rules for Attribution Map Generation
• Noise Modulation: Let Your Model Interpret Itself
• Evaluating Attribution Methods using White-Box LSTMs
• Interpretable Artificial Intelligence through the Lens of Feature Interaction
• Deep Learning-Based Detection of the Acute Respiratory Distress Syndrome: What Are the Models Learning?
• Attribution Mask: Filtering Out Irrelevant Features By Recursively Focusing Attention on Inputs of DNNs
• Gradient-Adjusted Neuron Activation Profiles for Comprehensive Introspection of Convolutional Speech Recognition Models
• Explainability Requires Interactivity
• Interpreting Deep Neural Networks with Relative Sectional Propagation by Analyzing Comparative Gradients and Hostile Activations
• Canonical Saliency Maps: Decoding Deep Face Models
• Sanity Simulations for Saliency Methods
• Play Fair: Frame Attributions in Video Models
• Distilling neural networks into skipgram-level decision lists
• A Neural Frequency-Severity Model and Its Application to Insurance Claims
• Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models
• Validation and generalization of pixel-wise relevance in convolutional neural networks trained for face classification
• Orthogonal Deep Models As Defense Against Black-Box Attacks
• Towards Explainable Fact Checking
• Deep learning for bioimage analysis
• IFBiD: Inference-Free Bias Detection
• A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines