Visualizing Deep Neural Network Decisions: Prediction Difference Analysis

Abstract

This article presents the prediction difference analysis method for visualizing the response of a deep neural network to a specific input. When classifying images, the method highlights areas in a given input image that provide evidence for or against a certain class. It overcomes several shortcoming of previous methods and provides great additional insight into the decision making process of classifiers. Making neural network decisions interpretable through visualization is important both to improve models and to accelerate the adoption of black-box classifiers in application areas such as medicine. We illustrate the method in experiments on natural images (ImageNet data), as well as medical images (MRI brain scans).

ICLR2017

References in corpus (4)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Understanding Neural Networks Through Deep Visualization
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences

Cited by in corpus (131)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• Learning Important Features Through Propagating Activation Differences
• On the Opportunities and Risks of Foundation Models
• Interpretable machine learning: definitions, methods, and applications
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
• Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
• Machine learning methods for histopathological image analysis
• GNNExplainer: Generating Explanations for Graph Neural Networks
• Towards Explainable Artificial Intelligence
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records
• Deformable ConvNets v2: More Deformable, Better Results
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions
• Explanations can be manipulated and geometry is to blame
• Explaining by Removing: A Unified Framework for Model Explanation
• Post-hoc explanation of black-box classifiers using confident itemsets
• Levels of explainable artificial intelligence for human-aligned conversational explanations
• Women also Snowboard: Overcoming Bias in Captioning Models
• DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
• Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
• TSViz: Demystification of Deep Learning Models for Time-Series Analysis
• Hierarchical interpretations for neural network predictions
• Explainable Artificial Intelligence: a Systematic Review
• Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examples
• Building and Interpreting Deep Similarity Models
• Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?
• A Survey of Deep Learning for Scientific Discovery
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation
• Interpreting deep learning models for weak lensing
• VisualBackProp: efficient visualization of CNNs
• On Interpretability of Artificial Neural Networks: A Survey
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• Saccader: Improving Accuracy of Hard Attention Models for Vision
• Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
• Detecting Out-of-Distribution Inputs in Deep Neural Networks Using an Early-Layer Output
• Dropout Feature Ranking for Deep Learning Models
• Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos
• Interpretable Graph Convolutional Neural Networks for Inference on Noisy Knowledge Graphs
• Visual Interpretability for Deep Learning: a Survey
• Autoencoder Node Saliency: Selecting Relevant Latent Representations
• A Survey on Neural Network Interpretability
• Opening the black box of neural nets: case studies in stop/top discrimination
• Visual pathways from the perspective of cost functions and multi-task deep neural networks
• Regional Multi-scale Approach for Visually Pleasing Explanations of Deep Neural Networks
• Towards Best Practice of Interpreting Deep Learning Models for EEG-based Brain Computer Interfaces
• Interpretability Beyond Classification Output: Semantic Bottleneck Networks
• Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
• Interpretable deep learning for nuclear deformation in heavy ion collisions
• Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information
• Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency
• Model-agnostic explainable artificial intelligence for object detection in image data
• Evaluating Weakly Supervised Object Localization Methods Right
• Saliency Methods for Explaining Adversarial Attacks
• Feature Removal Is a Unifying Principle for Model Explanation Methods
• Refining Language Models with Compositional Explanations
• Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
• Improving Interpretability of Deep Neural Networks with Semantic Information
• Deeply Explain CNN via Hierarchical Decomposition
• Evaluations and Methods for Explanation through Robustness Analysis
• Understanding Bias in Machine Learning
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case
• An Introduction to Deep Visual Explanation
• AUTOLYCUS: Exploiting Explainable AI (XAI) for Model Extraction Attacks against Interpretable Models
• Shapley explainability on the data manifold
• Towards Understanding Neural Machine Translation with Word Importance
• PredDiff: Explanations and Interactions from Conditional Expectations
• Being Bayesian about Categorical Probability
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• Brain Biomarker Interpretation in ASD Using Deep Learning and fMRI
• Self-explanatory Deep Salient Object Detection
• Relating Input Concepts to Convolutional Neural Network Decisions
• Transformation Importance with Applications to Cosmology
• LIMEcraft: Handcrafted superpixel selection and inspection for Visual eXplanations
• Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
• What evidence does deep learning model use to classify Skin Lesions?
• Dissecting Catastrophic Forgetting in Continual Learning by Deep Visualization
• Evaluating neural network explanation methods using hybrid documents and morphological agreement
• XRAI: Better Attributions Through Regions
• Human-interpretable model explainability on high-dimensional data
• Computationally Efficient Measures of Internal Neuron Importance
• Towards Interrogating Discriminative Machine Learning Models
• SISC: End-to-end Interpretable Discovery Radiomics-Driven Lung Cancer Prediction via Stacked Interpretable Sequencing Cells
• Interpretation of NLP models through input marginalization
• CAMERAS: Enhanced Resolution And Sanity preserving Class Activation Mapping for image saliency
• Information-Theoretic Visual Explanation for Black-Box Classifiers
• Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
• A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networks
• Aggregating explanation methods for stable and robust explainability
• Understanding Deep Architectures by Visual Summaries
• Evaluating Explanation Methods for Neural Machine Translation
• TSInsight: A local-global attribution framework for interpretability in time-series data
• Challenges for cognitive decoding using deep learning methods
• Contextual Prediction Difference Analysis for Explaining Individual Image Classifications
• Human Understandable Explanation Extraction for Black-box Classification Models Based on Matrix Factorization
• Efficient Image Evidence Analysis of CNN Classification Results
• Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions
• Ensemble learning with 3D convolutional neural networks for connectome-based prediction
• Decision Explanation and Feature Importance for Invertible Networks
• Capacity allocation analysis of neural networks: A tool for principled architecture design
• Using KL-divergence to focus Deep Visual Explanation
• Modeling Latent Attention Within Neural Networks
• Capacity allocation through neural network layers
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• Efficient Interpretation of Deep Learning Models Using Graph Structure and Cooperative Game Theory: Application to ASD Biomarker Discovery
• Grounding Visual Explanations
• Explaining How Deep Neural Networks Forget by Deep Visualization
• Understanding Character Recognition using Visual Explanations Derived from the Human Visual System and Deep Networks
• Generalization on the Enhancement of Layerwise Relevance Interpretability of Deep Neural Network
• Deep Learning Based Decision Support for Medicine -- A Case Study on Skin Cancer Diagnosis
• A General Taylor Framework for Unifying and Revisiting Attribution Methods
• Discriminative Attribution from Counterfactuals
• Simplifying the explanation of deep neural networks with sufficient and necessary feature-sets: case of text classification
• Understanding of Kernels in CNN Models by Suppressing Irrelevant Visual Features in Images
• DEPARA: Deep Attribution Graph for Deep Knowledge Transferability
• Detecting conflicting summary statistics in likelihood-free inference
• Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
• Balancing Robustness and Sensitivity using Feature Contrastive Learning
• An Overview of Computational Approaches for Interpretation Analysis
• Calibrating Class Activation Maps for Long-Tailed Visual Recognition
• Efficient Modelling Across Time of Human Actions and Interactions
• PREVIS -- A Combined Machine Learning and Visual Interpolation Approach for Interactive Reverse Engineering in Assembly Quality Control
• XAlgo: a Design Probe of Explaining Algorithms' Internal States via Question-Answering
• Deep Relevance Regularization: Interpretable and Robust Tumor Typing of Imaging Mass Spectrometry Data
• Explainable Deep Modeling of Tabular Data using TableGraphNet
• Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models
• SEEN: Sharpening Explanations for Graph Neural Networks using Explanations from Neighborhoods
• Rank Projection Trees for Multilevel Neural Network Interpretation