SmoothGrad: removing noise by adding noise

Abstract

Explaining the output of a deep network remains a challenge. In the case of an image classifier, one type of explanation is to identify pixels that strongly influence the final decision. A starting point for this strategy is the gradient of the class score function with respect to the input image. This gradient can be interpreted as a sensitivity map, and there are several techniques that elaborate on this basic idea. This paper makes two contributions: it introduces SmoothGrad, a simple method that can help visually sharpen gradient-based sensitivity maps, and it discusses lessons in the visualization of these maps. We publish the code for our experiments and a website with our results.

10 pages

References in corpus (1)

• Towards A Rigorous Science of Interpretable Machine Learning

Cited by in corpus (315)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• A Survey on the Explainability of Supervised Machine Learning
• Reliable Tuberculosis Detection using Chest X-ray with Deep Learning, Segmentation and Visualization
• Captum: A unified and generic model interpretability library for PyTorch
• Towards Explainable Artificial Intelligence
• This Looks Like That: Deep Learning for Interpretable Image Recognition
• The What-If Tool: Interactive Probing of Machine Learning Models
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Adversarial Examples Are Not Bugs, They Are Features
• Understanding the Role of Individual Units in a Deep Neural Network
• Analyzing Federated Learning through an Adversarial Lens
• Concept Whitening for Interpretable Image Recognition
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models
• Evaluating explainable artificial intelligence methods for multi-label deep learning classification tasks in remote sensing
• Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs
• Explanations can be manipulated and geometry is to blame
• Explaining by Removing: A Unified Framework for Model Explanation
• An Evaluation of the Human-Interpretability of Explanation
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• COV-ECGNET: COVID-19 detection using ECG trace images with deep convolutional neural network
• DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• Streaming convolutional neural networks for end-to-end learning with multi-megapixel images
• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• WT5?! Training Text-to-Text Models to Explain their Predictions
• Impossibility Theorems for Feature Attribution
• How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
• Hierarchical interpretations for neural network predictions
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• Understanding and Enhancing the Transferability of Adversarial Examples
• Explaining Classifiers with Causal Concept Effect (CaCE)
• Classification of crystallization outcomes using deep convolutional neural networks
• On Explainability of Graph Neural Networks via Subgraph Explorations
• Explainable Artificial Intelligence: a Systematic Review
• Benchmarking Attribution Methods with Relative Feature Importance
• Benchmarking Deep Learning Interpretability in Time Series Predictions
• Towards Automatic Concept-based Explanations
• Context-encoding Variational Autoencoder for Unsupervised Anomaly Detection
• Explainable Artificial Intelligence for Process Mining: A General Overview and Application of a Novel Local Explanation Approach for Predictive Process Monitoring
• Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Building and Interpreting Deep Similarity Models
• On the (In)fidelity and Sensitivity for Explanations
• Explainability Techniques for Graph Convolutional Networks
• A Survey of Deep Learning for Scientific Discovery
• Testing and verification of neural-network-based safety-critical control software: A systematic literature review
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• Restricting the Flow: Information Bottlenecks for Attribution
• Robust Local Features for Improving the Generalization of Adversarial Training
• SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localization
• On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors
• On Completeness-aware Concept-Based Explanations in Deep Neural Networks
• Temporal Graph Convolutional Networks for Automatic Seizure Detection
• Beyond saliency: understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Efficient Saliency Maps for Explainable AI
• On Interpretability of Artificial Neural Networks: A Survey
• When Explanations Lie: Why Many Modified BP Attributions Fail
• Regularizing Black-box Models for Improved Interpretability
• NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks
• Attention Branch Network: Learning of Attention Mechanism for Visual Explanation
• Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability
• Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks
• Neural Network Attributions: A Causal Perspective
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• Understanding Integrated Gradients with SmoothTaylor for Deep Neural Network Attribution
• Code Prediction by Feeding Trees to Transformers
• Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
• Towards falsifiable interpretability research
• iNNvestigate neural networks!
• The effectiveness of feature attribution methods and its correlation with automatic evaluation scores
• Explaining deep learning of galaxy morphology with saliency mapping
• Smoothed Geometry for Robust Attribution
• Bridging Adversarial Robustness and Gradient Interpretability
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Meta-trained agents implement Bayes-optimal agents
• Beyond Individualized Recourse: Interpretable and Interactive Summaries of Actionable Recourses
• Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
• Producing radiologist-quality reports for interpretable artificial intelligence
• Deep Stable Learning for Out-Of-Distribution Generalization
• SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation
• Issues with post-hoc counterfactual explanations: a discussion
• Saliency Methods for Explaining Adversarial Attacks
• A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges
• A Case for the Score: Identifying Image Anomalies using Variational Autoencoder Gradients
• Transferable Perturbations of Deep Feature Distributions
• Quantifying the Preferential Direction of the Model Gradient in Adversarial Training With Projected Gradient Descent
• Single Day Outdoor Photometric Stereo
• Feature Removal Is a Unifying Principle for Model Explanation Methods
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
• Do Concept Bottleneck Models Learn as Intended?
• Adversarial Infidelity Learning for Model Interpretation
• Interpretable Deep Learning under Fire
• xGEMs: Generating Examplars to Explain Black-Box Models
• Deep Learning in Protein Structural Modeling and Design
• Attention is Not Only a Weight: Analyzing Transformers with Vector Norms
• Interpreting Super-Resolution Networks with Local Attribution Maps
• Object-aware Contrastive Learning for Debiased Scene Representation
• On Baselines for Local Feature Attributions
• Towards Robust, Locally Linear Deep Networks
• Certifiably Robust Interpretation in Deep Learning
• Interpreting Deep Classifier by Visual Distillation of Dark Knowledge
• Explaining Bayesian Neural Networks
• A Rate-Distortion Framework for Explaining Neural Network Decisions
• Evaluations and Methods for Explanation through Robustness Analysis
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• Can I trust you more? Model-Agnostic Hierarchical Explanations
• Understanding Bias in Machine Learning
• A Peg-in-hole Task Strategy for Holes in Concrete
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case
• AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models
• Best of both worlds: local and global explanations with human-understandable concepts
• One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images
• Individual Explanations in Machine Learning Models: A Survey for Practitioners
• WQT and DG-YOLO: towards domain generalization in underwater object detection
• Deep Active Learning for Text Classification with Diverse Interpretations
• Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation
• Revisiting Sanity Checks for Saliency Maps
• Analysis of Deep Networks for Monocular Depth Estimation Through Adversarial Attacks with Proposal of a Defense Method
• MEGEX: Data-Free Model Extraction Attack against Gradient-Based Explainable AI
• Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
• X-ray Study of Spatial Structures in Tycho's Supernova Remnant Using Unsupervised Deep Learning
• Spatio-Temporal Perturbations for Video Attribution
• Discovering Molecular Functional Groups Using Graph Convolutional Neural Networks
• Generalizing multistain immunohistochemistry tissue segmentation using one-shot color deconvolution deep neural networks
• Exploring the Effect of Image Enhancement Techniques on COVID-19 Detection using Chest X-rays Images
• Interpreting Deep Neural Networks Through Variable Importance
• IROF: a low resource evaluation metric for explanation methods
• Enhancing Deep Neural Network Saliency Visualizations with Gradual Extrapolation
• Saliency-driven Word Alignment Interpretation for Neural Machine Translation
• Visualization of Convolutional Neural Networks for Monocular Depth Estimation
• OrigamiNet: Weakly-Supervised, Segmentation-Free, One-Step, Full Page Text Recognition by learning to unfold
• Trustworthy Convolutional Neural Networks: A Gradient Penalized-based Approach
• Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification
• Incorporating Priors with Feature Attribution on Text Classification
• A simple defense against adversarial attacks on heatmap explanations
• Algorithmic Recourse in the Wild: Understanding the Impact of Data and Model Shifts
• PointMask: Towards Interpretable and Bias-Resilient Point Cloud Processing
• Oblique Decision Trees from Derivatives of ReLU Networks
• A Baseline for Shapley Values in MLPs: from Missingness to Neutrality
• Debugging Tests for Model Explanations
• Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
• XProtoNet: Diagnosis in Chest Radiography with Global and Local Explanations
• Semantics for Global and Local Interpretation of Deep Neural Networks
• Dynamically Sampled Nonlocal Gradients for Stronger Adversarial Attacks
• XRAI: Better Attributions Through Regions
• Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps
• Adversarial Attacks and Defenses: An Interpretation Perspective
• MeLIME: Meaningful Local Explanation for Machine Learning Models
• Evaluating Saliency Methods for Neural Language Models
• Interpretable Graph Capsule Networks for Object Recognition
• CAMERAS: Enhanced Resolution And Sanity preserving Class Activation Mapping for image saliency
• Usefulness of interpretability methods to explain deep learning based plant stress phenotyping
• Robust Models Are More Interpretable Because Attributions Look Normal
• Maximally Invariant Data Perturbation as Explanation
• X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust
• Information-Theoretic Visual Explanation for Black-Box Classifiers
• Preserve, Promote, or Attack? GNN Explanation via Topology Perturbation
• Interpreting Interpretations: Organizing Attribution Methods by Criteria
• Can We Trust Your Explanations? Sanity Checks for Interpreters in Android Malware Analysis
• A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networks
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• Improving Interpretability in Medical Imaging Diagnosis using Adversarial Training
• The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
• Robust and Stable Black Box Explanations
• Deep Neural Networks for Choice Analysis: Extracting Complete Economic Information for Interpretation
• Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsity
• Fooling Object Detectors: Adversarial Attacks by Half-Neighbor Masks
• Towards the Unification and Robustness of Perturbation and Gradient Based Explanations
• Aggregating explanation methods for stable and robust explainability
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• TSInsight: A local-global attribution framework for interpretability in time-series data
• Generative Counterfactuals for Neural Networks via Attribute-Informed Perturbation
• Multi-Stage Influence Function
• Feature Attributions and Counterfactual Explanations Can Be Manipulated
• Towards Robust Explanations for Deep Neural Networks
• Passive Attention in Artificial Neural Networks Predicts Human Visual Selectivity
• Gradient-based Analysis of NLP Models is Manipulable
• How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels
• A Review of Explainable Artificial Intelligence in Manufacturing
• Challenges for cognitive decoding using deep learning methods
• An Empirical Study of Explainable AI Techniques on Deep Learning Models For Time Series Tasks
• DNN2LR: Interpretation-inspired Feature Crossing for Real-world Tabular Data
• Concept Learners for Few-Shot Learning
• Towards Interpretable Ensemble Learning for Image-based Malware Detection
• Contextual Prediction Difference Analysis for Explaining Individual Image Classifications
• Are Interpretations Fairly Evaluated? A Definition Driven Pipeline for Post-Hoc Interpretability
• Sequential Explanations with Mental Model-Based Policies
• DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
• Evaluation, Tuning and Interpretation of Neural Networks for Meteorological Applications
• Large scale evaluation of importance maps in automatic speech recognition
• Attention Flows are Shapley Value Explanations
• GANMEX: One-vs-One Attributions Guided by GAN-based Counterfactual Explanation Baselines
• Leveraging Sparse Linear Layers for Debuggable Deep Networks
• WHO 2016 subtyping and automated segmentation of glioma using multi-task deep learning
• Towards Visually Explaining Video Understanding Networks with Perturbation
• Quantitative and Qualitative Evaluation of Explainable Deep Learning Methods for Ophthalmic Diagnosis
• Counterfactual Generation with Knockoffs
• DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation
• On a Sparse Shortcut Topology of Artificial Neural Networks
• Convolutional Neural Network Models and Interpretability for the Anisotropic Reynolds Stress Tensor in Turbulent One-dimensional Flows
• Towards Rigorous Interpretations: a Formalisation of Feature Attribution
• Intelligence plays dice: Stochasticity is essential for machine learning
• Born Identity Network: Multi-way Counterfactual Map Generation to Explain a Classifier's Decision
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• XDeep: An Interpretation Tool for Deep Neural Networks
• The Impact of Feature Representation on the Accuracy of Photonic Neural Networks
• Crowdsourcing Evaluation of Saliency-based XAI Methods
• CXR-Net: An Artificial Intelligence Pipeline for Quick Covid-19 Screening of Chest X-Rays
• MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
• Towards Human-Understandable Visual Explanations:Imperceptible High-frequency Cues Can Better Be Removed
• What's wrong with this video? Comparing Explainers for Deepfake Detection
• SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers
• An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness
• Regularizing Black-box Models for Improved Interpretability (HILL 2019 Version)
• Explaining neural network predictions of material strength
• Combining Similarity and Adversarial Learning to Generate Visual Explanation: Application to Medical Image Classification
• Measuring and improving the quality of visual explanations
• What Do Adversarially Robust Models Look At?
• A Step Towards Exposing Bias in Trained Deep Convolutional Neural Network Models
• Noise Optimization for Artificial Neural Networks
• Evaluation of Saliency-based Explainability Method
• Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning
• Explainable Recommender Systems via Resolving Learning Representations
• QUACKIE: A NLP Classification Task With Ground Truth Explanations
• TIME: A Transparent, Interpretable, Model-Adaptive and Explainable Neural Network for Dynamic Physical Processes
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• Explaining Regression Based Neural Network Model
• Towards Explanation of DNN-based Prediction with Guided Feature Inversion
• ExAD: An Ensemble Approach for Explanation-based Adversarial Detection
• Adversarial TCAV -- Robust and Effective Interpretation of Intermediate Layers in Neural Networks
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms
• Shapley Explanation Networks
• Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution
• Weakly Supervised Minirhizotron Image Segmentation with MIL-CAM
• Identifying Pediatric Vascular Anomalies With Deep Learning
• Learning by Self-Explanation, with Application to Neural Architecture Search
• Debiasing Convolutional Neural Networks via Meta Orthogonalization
• Discriminative Attribution from Counterfactuals
• CIM: Class-Irrelevant Mapping for Few-Shot Classification
• Harnessing value from data science in business: ensuring explainability and fairness of solutions
• Evaluating the Use of Reconstruction Error for Novelty Localization
• Towards Understanding the Effectiveness of Attention Mechanism
• Inverting and Understanding Object Detectors
• Contrastive ACE: Domain Generalization Through Alignment of Causal Mechanisms
• Bounded logit attention: Learning to explain image classifiers
• Gradient Frequency Modulation for Visually Explaining Video Understanding Models
• ExCon: Explanation-driven Supervised Contrastive Learning for Image Classification
• Where and When: Space-Time Attention for Audio-Visual Explanations
• A General Taylor Framework for Unifying and Revisiting Attribution Methods
• EDDA: Explanation-driven Data Augmentation to Improve Explanation Faithfulness
• Building Reliable Explanations of Unreliable Neural Networks: Locally Smoothing Perspective of Model Interpretation
• Do Input Gradients Highlight Discriminative Features?
• What do CNN neurons learn: Visualization & Clustering
• Scaling Symbolic Methods using Gradients for Neural Model Explanation
• Assessing the Reliability of Visual Explanations of Deep Models with Adversarial Perturbations
• DANCE: Enhancing saliency maps using decoys
• A Programmatic and Semantic Approach to Explaining and DebuggingNeural Network Based Object Detectors
• Weakly-Supervised Cell Tracking via Backward-and-Forward Propagation
• NeuroMask: Explaining Predictions of Deep Neural Networks through Mask Learning
• Pattern-Guided Integrated Gradients
• Introspective Learning by Distilling Knowledge from Online Self-explanation
• Visualizing Color-wise Saliency of Black-Box Image Classification Models
• Interpretable Disentanglement of Neural Networks by Extracting Class-Specific Subnetwork
• Learning Preference-Based Similarities from Face Images using Siamese Multi-Task CNNs
• BayesGrad: Explaining Predictions of Graph Convolutional Networks
• Beyond Explainability: Leveraging Interpretability for Improved Adversarial Learning
• Explainability Requires Interactivity
• AutoToon: Automatic Geometric Warping for Face Cartoon Generation
• Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution
• ST-ABN: Visual Explanation Taking into Account Spatio-temporal Information for Video Recognition
• Reconstructing Actions To Explain Deep Reinforcement Learning
• Learning Propagation Rules for Attribution Map Generation
• Deep Synoptic Monte Carlo Planning in Reconnaissance Blind Chess
• Mutual Information Preserving Back-propagation: Learn to Invert for Faithful Attribution
• Influence Patterns for Explaining Information Flow in BERT
• Inserting Information Bottlenecks for Attribution in Transformers
• Weakly Supervised Recovery of Semantic Attributes
• Exploring the Role of BERT Token Representations to Explain Sentence Probing Results
• Improving Attribution Methods by Learning Submodular Functions
• Signed Input Regularization
• Noise Modulation: Let Your Model Interpret Itself
• Diagnostic Visualization for Deep Neural Networks Using Stochastic Gradient Langevin Dynamics
• Canonical Saliency Maps: Decoding Deep Face Models
• Defense Against Explanation Manipulation
• How to Explain Neural Networks: an Approximation Perspective
• Sanity Simulations for Saliency Methods
• $\P$ILCRO: Making Importance Landscapes Flat Again
• CACTUS: Detecting and Resolving Conflicts in Objective Functions
• Simulating time to event prediction with spatiotemporal echocardiography deep learning
• Training Machine Learning Models by Regularizing their Explanations
• ProtoShotXAI: Using Prototypical Few-Shot Architecture for Explainable AI
• DNN2LR: Automatic Feature Crossing for Credit Scoring
• Determining the Dark Matter distribution in galaxies with Deep Learning
• Robustness of different loss functions and their impact on networks learning capability
• A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines
• Towards Auditability for Fairness in Deep Learning
• Robust Domain-Free Domain Generalization with Class-aware Alignment
• Explainable Deep Reinforcement Learning for Portfolio Management: An Empirical Approach
• Balancing Robustness and Sensitivity using Feature Contrastive Learning
• Semantic Network Interpretation
• Gradient-Based Interpretability Methods and Binarized Neural Networks
• Analyzing Representations inside Convolutional Neural Networks
• Automated Dependence Plots
• Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems
• Localizing Interpretable Multi-scale informative Patches Derived from Media Classification Task
• Deep Active Learning by Model Interpretability
• Attention Weights in Transformer NMT Fail Aligning Words Between Sequences but Largely Explain Model Predictions
• A copula-based visualization technique for a neural network
• Games for Fairness and Interpretability