Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization

Abstract

We propose a technique for producing "visual explanations" for decisions from a large class of CNN-based models, making them more transparent. Our approach - Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept, flowing into the final convolutional layer to produce a coarse localization map highlighting important regions in the image for predicting the concept. Grad-CAM is applicable to a wide variety of CNN model-families: (1) CNNs with fully-connected layers, (2) CNNs used for structured outputs, (3) CNNs used in tasks with multimodal inputs or reinforcement learning, without any architectural changes or re-training. We combine Grad-CAM with fine-grained visualizations to create a high-resolution class-discriminative visualization and apply it to off-the-shelf image classification, captioning, and visual question answering (VQA) models, including ResNet-based architectures. In the context of image classification models, our visualizations (a) lend insights into their failure modes, (b) are robust to adversarial images, (c) outperform previous methods on localization, (d) are more faithful to the underlying model and (e) help achieve generalization by identifying dataset bias. For captioning and VQA, we show that even non-attention based models can localize inputs. We devise a way to identify important neurons through Grad-CAM and combine it with neuron names to provide textual explanations for model decisions. Finally, we design and conduct human studies to measure if Grad-CAM helps users establish appropriate trust in predictions from models and show that Grad-CAM helps untrained users successfully discern a 'stronger' nodel from a 'weaker' one even when both make identical predictions. Our code is available at https://github.com/ramprs/grad-cam/, along with a demo at http://gradcam.cloudcv.org, and a video at youtu.be/COjUB9Izk6E.

This version was published in International Journal of Computer Vision (IJCV) in 2019; A previous version of the paper was published at International Conference on Computer Vision (ICCV'17)

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Striving for Simplicity: The All Convolutional Net
• Hierarchical Question-Image Co-Attention for Visual Question Answering

Cited by in corpus (184)

• Deep Learning for Chest X-ray Analysis: A Survey
• Explainable Artificial Intelligence Applications in Cyber Security: State-of-the-Art in Research
• Detecting Anemia from Retinal Fundus Images
• Human Activity Recognition Using Tools of Convolutional Neural Networks: A State of the Art Review, Data Sets, Challenges and Future Prospects
• YOLO-MS: Rethinking Multi-Scale Representation Learning for Real-time Object Detection
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Adversarial attacks and defenses in explainable artificial intelligence: A survey
• XCM: An Explainable Convolutional Neural Network for Multivariate Time Series Classification
• Multi-Granularity Canonical Appearance Pooling for Remote Sensing Scene Classification
• Class-Incremental Continual Learning into the eXtended DER-verse
• Ethics of Artificial Intelligence and Robotics in the Architecture, Engineering, and Construction Industry
• Explainable artificial intelligence in breast cancer detection and risk prediction: A systematic scoping review
• RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
• COV-ECGNET: COVID-19 detection using ECG trace images with deep convolutional neural network
• Perspectives on individual animal identification from biology and computer vision
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Experimental velocity data estimation for imperfect particle images using machine learning
• A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
• Explainable Diabetic Retinopathy Detection and Retinal Image Generation
• ExoMiner: A Highly Accurate and Explainable Deep Learning Classifier that Validates 301 New Exoplanets
• Predicting post-operative right ventricular failure using video-based deep learning
• Boosting Fast Adversarial Training with Learnable Adversarial Initialization
• Explainable AI, but explainable to whom?
• A Real-World Demonstration of Machine Learning Generalizability: Intracranial Hemorrhage Detection on Head CT
• XEM: An Explainable-by-Design Ensemble Method for Multivariate Time Series Classification
• Seabed classification using physics-based modeling and machine learning
• Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps
• Medical Imaging and Machine Learning
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model
• Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization
• SAAN: Similarity-aware attention flow network for change detection with VHR remote sensing images
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• DeepMerge II: Building Robust Deep Learning Algorithms for Merging Galaxy Identification Across Domains
• Analysis of a Deep Learning Model for 12-Lead ECG Classification Reveals Learned Features Similar to Diagnostic Criteria
• Exploring convolutional neural networks with transfer learning for diagnosing Lyme disease from skin lesion images
• GANs and alternative methods of synthetic noise generation for domain adaption of defect classification of Non-destructive ultrasonic testing
• Deep Learning Predicts Prevalent and Incident Parkinson's Disease From UK Biobank Fundus Imaging
• Intra-Domain Task-Adaptive Transfer Learning to Determine Acute Ischemic Stroke Onset Time
• A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application
• Self-learning for weakly supervised Gleason grading of local patterns
• A comprehensive review of 3D convolutional neural network-based classification techniques of diseased and defective crops using non-UAV-based hyperspectral images
• EXMOS: Explanatory Model Steering Through Multifaceted Explanations and Data Configurations
• Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
• DuetFace: Collaborative Privacy-Preserving Face Recognition via Channel Splitting in the Frequency Domain
• Artificial Intelligence and Diabetes Mellitus: An Inside Look Through the Retina
• Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention
• Innovative Speech-Based Deep Learning Approaches for Parkinson's Disease Classification: A Systematic Review
• Understanding Integrated Gradients with SmoothTaylor for Deep Neural Network Attribution
• White Box Methods for Explanations of Convolutional Neural Networks in Image Classification Tasks
• Quantum Capsule Networks
• Regional Multi-scale Approach for Visually Pleasing Explanations of Deep Neural Networks
• Entanglement-guided architectures of machine learning by quantum tensor network
• Explaining Human Activity Recognition with SHAP: Validating Insights with Perturbation and Quantitative Measures
• Autoencoding Galaxy Spectra I: Architecture
• Survey on Hand Gesture Recognition from Visual Input
• Deep Learning for Identifying Iran's Cultural Heritage Buildings in Need of Conservation Using Image Classification and Grad-CAM
• Sparse Visual Counterfactual Explanations in Image Space
• Deep learning approach for identification of HII regions during reionization in 21-cm observations -- II. foreground contamination
• Model-agnostic explainable artificial intelligence for object detection in image data
• SLISEMAP: Supervised dimensionality reduction through local explanations
• Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation
• Biomarker Investigation using Multiple Brain Measures from MRI through XAI in Alzheimer's Disease Classification
• ActSonic: Recognizing Everyday Activities from Inaudible Acoustic Wave Around the Body
• Machine learning approaches for automatic defect detection in photovoltaic systems
• Ubi-SleepNet: Advanced Multimodal Fusion Techniques for Three-stage Sleep Classification Using Ubiquitous Sensing
• FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing
• (ASNA) An Attention-based Siamese-Difference Neural Network with Surrogate Ranking Loss function for Perceptual Image Quality Assessment
• Convolutional neural network based decoders for surface codes
• Automated pharyngeal phase detection and bolus localization in videofluoroscopic swallowing study: Killing two birds with one stone?
• Deeply Explain CNN via Hierarchical Decomposition
• Explainable AI: XAI-Guided Context-Aware Data Augmentation
• Learning at a Glance: Towards Interpretable Data-limited Continual Semantic Segmentation via Semantic-Invariance Modelling
• An Explainable Contrastive-based Dilated Convolutional Network with Transformer for Pediatric Pneumonia Detection
• Feature Visualization within an Automated Design Assessment leveraging Explainable Artificial Intelligence Methods
• Occlusion Sensitivity Analysis with Augmentation Subspace Perturbation in Deep Feature Space
• Fooling Partial Dependence via Data Poisoning
• Neural Transformers for Intraductal Papillary Mucosal Neoplasms (IPMN) Classification in MRI images
• Deep Active Learning for Text Classification with Diverse Interpretations
• Plasma Image Classification Using Cosine Similarity Constrained CNN
• Activation Landscapes as a Topological Summary of Neural Network Performance
• Deep Recursive Embedding for High-Dimensional Data
• TSGB: Target-Selective Gradient Backprop for Probing CNN Visual Saliency
• Classification and reconstruction of optical quantum states with deep neural networks
• Resource-Frugal Classification and Analysis of Pathology Slides Using Image Entropy
• LimitNet: Progressive, Content-Aware Image Offloading for Extremely Weak Devices & Networks
• O'TRAIN: a robust and flexible Real/Bogus classifier for the study of the optical transient sky
• Physics-Assisted Reduced-Order Modeling for Identifying Dominant Features of Transonic Buffet
• Incorporating Anatomical Awareness for Enhanced Generalizability and Progression Prediction in Deep Learning-Based Radiographic Sacroiliitis Detection
• Automatic Bat Call Classification using Transformer Networks
• From explanation to synthesis: Compositional program induction for learning from demonstration
• Explaining Predictive Uncertainty by Exposing Second-Order Effects
• Weakly Supervised Attention Model for RV StrainClassification from volumetric CTPA Scans
• TEGLIE: Transformer encoders as strong gravitational lens finders in KiDS
• Optimising Knee Injury Detection with Spatial Attention and Validating Localisation Ability
• Choose Your Explanation: A Comparison of SHAP and GradCAM in Human Activity Recognition
• Identifying the Defective: Detecting Damaged Grains for Cereal Appearance Inspection
• LioNets: A Neural-Specific Local Interpretation Technique Exploiting Penultimate Layer Information
• Shaping Visual Representations with Attributes for Few-Shot Recognition
• Identification of Rare Cortical Folding Patterns using Unsupervised Deep Learning
• Dermoscopic Dark Corner Artifacts Removal: Friend or Foe?
• Generalized Semantic Contrastive Learning via Embedding Side Information for Few-Shot Object Detection
• Semantic interpretation for convolutional neural networks: What makes a cat a cat?
• LIMEcraft: Handcrafted superpixel selection and inspection for Visual eXplanations
• Rethinking Degradation: Radiograph Super-Resolution via AID-SRGAN
• Brain informed transfer learning for categorizing construction hazards
• Using Feature Alignment Can Improve Clean Average Precision and Adversarial Robustness in Object Detection
• Wise-SrNet: A Novel Architecture for Enhancing Image Classification by Learning Spatial Resolution of Feature Maps
• Transfer learning, alternative approaches, and visualization of a convolutional neural network for retrieval of the internuclear distance in a molecule from photoelectron momentum distributions
• Exploiting CLIP-based Multi-modal Approach for Artwork Classification and Retrieval
• Learning Personal Style from Few Examples
• Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
• Graph neural network-based structural classification of glass-forming liquids and its interpretation via self-attention mechanism
• Transfer Learning and Explainable AI for Brain Tumor Classification: A Study Using MRI Data from Bangladesh
• Towards Explaining Satellite Based Poverty Predictions with Convolutional Neural Networks
• Enhancing Cross-Dataset Performance of Distracted Driving Detection With Score Softmax Classifier And Dynamic Gaussian Smoothing Supervision
• CAMEL2: Enhancing weakly supervised learning for histopathology images by incorporating the significance ratio
• No Glitch in the Matrix: Robust Reconstruction of Gravitational Wave Signals Under Noise Artifacts
• Visual explanations of machine learning model estimating charge states in quantum dots
• A Trustworthiness Score to Evaluate DNN Predictions
• Chest X-Rays Image Classification from beta-Variational Autoencoders Latent Features
• Advancing Attribution-Based Neural Network Explainability through Relative Absolute Magnitude Layer-Wise Relevance Propagation and Multi-Component Evaluation
• Discriminative Feature Learning through Feature Distance Loss
• Solving the enigma: Enhancing faithfulness and comprehensibility in explanations of deep networks
• StegaFFD: Privacy-Preserving Face Forgery Detection via Fine-Grained Steganographic Domain Lifting
• Deep learning of topological phase transitions from entanglement aspects for two-dimensional chiral p-wave superconductors
• IMAGO: A family photo album dataset for a socio-historical analysis of the twentieth century
• Characterizing out-of-distribution generalization of neural networks: application to the disordered Su-Schrieffer-Heeger model
• Expediting DECam Multimessenger Counterpart Searches with Convolutional Neural Networks
• Attaining entropy production and dissipation maps from Brownian movies via neural networks
• Dreaming of Electrical Waves: Generative Modeling of Cardiac Excitation Waves using Diffusion Models
• A welding penetration prediction model for laser welding process based on self-supervised learning using physics-informed neural networks
• XInsight: Revealing Model Insights for GNNs with Flow-based Explanations
• Disentangled representations: towards interpretation of sex determination from hip bone
• Explainable Deep Learning in Medical Imaging: Brain Tumor and Pneumonia Detection
• From Visual Explanations to Counterfactual Explanations with Latent Diffusion
• GCAN: Generative Counterfactual Attention-guided Network for Explainable Cognitive Decline Diagnostics based on fMRI Functional Connectivity
• Brain Age Estimation with a Greedy Dual-Stream Model for Limited Datasets
• Unveiling Molecular Moieties through Hierarchical Grad-CAM Graph Explainability
• Pre or Post-Softmax Scores in Gradient-based Attribution Methods, What is Best?
• From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing
• Mode visualisation and control of complex lasers using neural networks
• Introducing DEFORMISE: A deep learning framework for dementia diagnosis in the elderly using optimized MRI slice selection
• ViG-Bias: Visually Grounded Bias Discovery and Mitigation
• X-ray2CTPA: Leveraging Diffusion Models to Enhance Pulmonary Embolism Classification
• Explainable AI: Context-Aware Layer-Wise Integrated Gradients for Explaining Transformer Models
• H-SemiS: Hierarchical Fusion of Semi and Self-Supervised Learning for Knee Osteoarthritis Severity Grading
• Wafer Defect Root Cause Analysis with Partial Trajectory Regression
• Progress in deep Markov State Modeling: Coarse graining and experimental data restraints
• Enhancing Exchange Rate Forecasting with Explainable Deep Learning Models
• Which Neurons Matter in IR? Applying Integrated Gradients-based Methods to Understand Cross-Encoders
• Leveraging Activation Maximization and Generative Adversarial Training to Recognize and Explain Patterns in Natural Areas in Satellite Imagery
• The Role of Pleura and Adipose in Lung Ultrasound AI
• On Spectral Properties of Gradient-based Explanation Methods
• Semi-supervised Learning From Demonstration Through Program Synthesis: An Inspection Robot Case Study
• Detection of Vascular Leukoencephalopathy in CT Images
• Improving Group Robustness on Spurious Correlation via Evidential Alignment
• Leveraging Expert Input for Robust and Explainable AI-Assisted Lung Cancer Detection in Chest X-rays
• Space-scale Exploration of the Poor Reliability of Deep Learning Models: the Case of the Remote Sensing of Rooftop Photovoltaic Systems
• Comprehensive Evaluation of Prototype Neural Networks
• MvHo-IB: Multi-View Higher-Order Information Bottleneck for Brain Disorder Diagnosis
• Reveal of Vision Transformers Robustness against Adversarial Attacks
• Explainability via Interactivity? Supporting Nonexperts' Sensemaking of Pretrained CNN by Interacting with Their Daily Surroundings
• ROIsGAN: A Region Guided Generative Adversarial Framework for Murine Hippocampal Subregion Segmentation
• ProtoTSNet: Interpretable Multivariate Time Series Classification With Prototypical Parts
• Segmenting proto-halos with vision transformers
• HOLMES: HOLonym-MEronym based Semantic inspection for Convolutional Image Classifiers
• Variance-Based Defense Against Blended Backdoor Attacks
• IMPACTX: improving model performance by appropriately constraining the training with teacher explanations
• Interpretable Prediction of Lymph Node Metastasis in Rectal Cancer MRI Using Variational Autoencoders
• Trust Through Transparency: Explainable Social Navigation for Autonomous Mobile Robots via Vision-Language Models
• What makes a steady flow to favour kinematic magnetic field generation: A statistical analysis
• We Can Always Catch You: Detecting Adversarial Patched Objects WITH or WITHOUT Signature
• Trustworthy AI-based crack-tip segmentation using domain-guided explanations
• M6: Multi-generator, Multi-domain, Multi-lingual and cultural, Multi-genres, Multi-instrument Machine-Generated Music Detection Databases
• CLAIRE-DSA: Fluoroscopic Image Classification for Quality Assurance of Computer Vision Pipelines in Acute Ischemic Stroke
• CNN-based Approaches For Cross-Subject Classification in Motor Imagery: From The State-of-The-Art to DynamicNet
• Constrained unsupervised anomaly segmentation
• CoPA: Hierarchical Concept Prompting and Aggregating Network for Explainable Diagnosis
• Identifying lopsidedness in spiral galaxies using a Deep Convolutional Neural Network
• Beyond Occlusion: In Search for Near Real-Time Explainability of CNN-Based Prostate Cancer Classification
• Interpretable Quantile Regression by Optimal Decision Trees
• LightTeaNet: A Weakly Supervised Lightweight CNN for Multi-Label Tea Leaf Disease Detection and Localization
• Explainable-by-Design Audio Deepfake Detection via Wiener-Hopf Linear Prediction