Learning Important Features Through Propagating Activation Differences

Abstract

The purported "black box" nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Deep Learning Important FeaTures), a method for decomposing the output prediction of a neural network on a specific input by backpropagating the contributions of all neurons in the network to every feature of the input. DeepLIFT compares the activation of each neuron to its 'reference activation' and assigns contribution scores according to the difference. By optionally giving separate consideration to positive and negative contributions, DeepLIFT can also reveal dependencies which are missed by other approaches. Scores can be computed efficiently in a single backward pass. We apply DeepLIFT to models trained on MNIST and simulated genomic data, and show significant advantages over gradient-based methods. Video tutorial: http://goo.gl/qKb7pL, ICML slides: bit.ly/deeplifticmlslides, ICML talk: https://vimeo.com/238275076, code: http://goo.gl/RM8jvH.

Updated to include changes present in the ICML camera-ready paper, and other small corrections

References in corpus (8)

• Striving for Simplicity: The All Convolutional Net
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Visualizing and Understanding Convolutional Networks
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• An unexpected unity among methods for interpreting model predictions
• Investigating the influence of noise and distractors on the interpretation of neural networks
• Gradients of Counterfactuals

Cited by in corpus (450)

• A Unified Approach to Interpreting Model Predictions
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Axiomatic Attribution for Deep Networks
• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• On the Opportunities and Risks of Foundation Models
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Explaining Explanations in AI
• GNNExplainer: Generating Explanations for Graph Neural Networks
• Captum: A unified and generic model interpretability library for PyTorch
• Towards Explainable Artificial Intelligence
• Learning data driven discretizations for partial differential equations
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Eigen-CAM: Class Activation Map using Principal Components
• Towards Robust Interpretability with Self-Explaining Neural Networks
• Explanations in Autonomous Driving: A Survey
• Explaining a Series of Models by Propagating Shapley Values
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• Polyconvex anisotropic hyperelasticity with neural networks
• Deep Learning in Pharmacogenomics: From Gene Regulation to Patient Stratification
• Technical Note on Transcription Factor Motif Discovery from Importance Scores (TF-MoDISco) version 0.5.6.5
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Evaluating explainable artificial intelligence methods for multi-label deep learning classification tasks in remote sensing
• Smooth Grad-CAM++: An Enhanced Inference Level Visualization Technique for Deep Convolutional Neural Network Models
• Enhancing Discrete Choice Models with Representation Learning
• XCM: An Explainable Convolutional Neural Network for Multivariate Time Series Classification
• Automatic Sleep Staging of EEG Signals: Recent Development, Challenges, and Future Directions
• Explanations can be manipulated and geometry is to blame
• Interpreting Blackbox Models via Model Extraction
• Evaluation of post-hoc interpretability methods in time-series classification
• Post-hoc explanation of black-box classifiers using confident itemsets
• On the Explainability of Natural Language Processing Deep Models
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• DeepAID: Interpreting and Improving Deep Learning-based Anomaly Detection in Security Applications
• Machine learning and AI-based approaches for bioactive ligand discovery and GPCR-ligand recognition
• From Clustering to Cluster Explanations via Neural Networks
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
• TimeSHAP: Explaining Recurrent Models through Sequence Perturbations
• TSViz: Demystification of Deep Learning Models for Time-Series Analysis
• ISeeU: Visually interpretable deep learning for mortality prediction inside the ICU
• Exploring Interpretable LSTM Neural Networks over Multi-Variable Data
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• Feature selection revisited in the single-cell era
• How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations
• Explainable Artificial Intelligence: a Systematic Review
• To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods
• Benchmarking Deep Learning Interpretability in Time Series Predictions
• Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience
• Explainable Diabetic Retinopathy Detection and Retinal Image Generation
• Explaining Anomalies Detected by Autoencoders Using SHAP
• Interpretable deep learning for guided structure-property explorations in photovoltaics
• Deep neural network improves the estimation of polygenic risk scores for breast cancer
• A general framework for inference on algorithm-agnostic variable importance
• XAIR: A Framework of Explainable AI in Augmented Reality
• An analysis on the use of autoencoders for representation learning: fundamentals, learning task case studies, explainability and challenges
• Interpretation of Neural Networks is Fragile
• "Why Should You Trust My Explanation?" Understanding Uncertainty in LIME Explanations
• Explainable Machine Learning for Public Policy: Use Cases, Gaps, and Research Directions
• Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning
• Deep Learning Based Cloud Cover Parameterization for ICON
• Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
• Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements?
• Neuron Shapley: Discovering the Responsible Neurons
• On the (In)fidelity and Sensitivity for Explanations
• Explainability Techniques for Graph Convolutional Networks
• Security and Privacy Issues in Deep Learning
• Explainability: Relevance based Dynamic Deep Learning Algorithm for Fault Detection and Diagnosis in Chemical Processes
• A Survey of Deep Learning for Scientific Discovery
• DIG: A Turnkey Library for Diving into Graph Deep Learning Research
• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• Testing and verification of neural-network-based safety-critical control software: A systematic literature review
• A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations
• A Survey of Safety and Trustworthiness of Deep Neural Networks: Verification, Testing, Adversarial Attack and Defence, and Interpretability
• Restricting the Flow: Information Bottlenecks for Attribution
• Medical Imaging and Machine Learning
• SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localization
• Explaining the Explainer: A First Theoretical Analysis of LIME
• Beyond saliency: understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation
• Towards Evaluating Explanations of Vision Transformers for Medical Imaging
• RRWaveNet: A Compact End-to-End Multi-Scale Residual CNN for Robust PPG Respiratory Rate Estimation
• Causally-informed deep learning to improve climate models and projections
• Parkinson's Disease Recognition Using SPECT Image and Interpretable AI: A Tutorial
• When Explanations Lie: Why Many Modified BP Attributions Fail
• PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks
• Towards Better Understanding Attribution Methods
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• Shedding Light on Black Box Machine Learning Algorithms: Development of an Axiomatic Framework to Assess the Quality of Methods that Explain Individual Predictions
• Can I Trust the Explainer? Verifying Post-hoc Explanatory Methods
• Do Not Trust Additive Explanations
• Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability
• How does this interaction affect me? Interpretable attribution for feature interactions
• Dropout Feature Ranking for Deep Learning Models
• A new interpretable unsupervised anomaly detection method based on residual explanation
• RIDDLE: Race and ethnicity Imputation from Disease history with Deep LEarning
• Considerations When Learning Additive Explanations for Black-Box Models
• Neural Network Attributions: A Causal Perspective
• Epistemic values in feature importance methods: Lessons from feminist epistemology
• Autoencoder Node Saliency: Selecting Relevant Latent Representations
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• Interpreting Black Box Models via Hypothesis Testing
• Explain Your Move: Understanding Agent Actions Using Specific and Relevant Feature Attribution
• A Survey on Neural Network Interpretability
• Granger-causal Attentive Mixtures of Experts: Learning Important Features with Neural Networks
• Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
• Understanding Integrated Gradients with SmoothTaylor for Deep Neural Network Attribution
• On the Art and Science of Machine Learning Explanations
• Towards falsifiable interpretability research
• Exploration of Interpretability Techniques for Deep COVID-19 Classification using Chest X-ray Images
• White Box Methods for Explanations of Convolutional Neural Networks in Image Classification Tasks
• Shapley Flow: A Graph-based Approach to Interpreting Model Predictions
• Inseq: An Interpretability Toolkit for Sequence Generation Models
• Global Aggregations of Local Explanations for Black Box models
• Regional Multi-scale Approach for Visually Pleasing Explanations of Deep Neural Networks
• SHAP values for Explaining CNN-based Text Classification Models
• Counterfactual Explanation Based on Gradual Construction for Deep Networks
• Explaining Clinical Decision Support Systems in Medical Imaging using Cycle-Consistent Activation Maximization
• Interpretability Beyond Classification Output: Semantic Bottleneck Networks
• Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
• Smoothed Geometry for Robust Attribution
• Explaining Time Series Predictions with Dynamic Masks
• Bridging Adversarial Robustness and Gradient Interpretability
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Self-Explaining Structures Improve NLP Models
• Investigating the significance of adversarial attacks and their relation to interpretability for radar-based human activity recognition systems
• Towards Frequency-Based Explanation for Robust CNN
• Infusing domain knowledge in AI-based "black box" models for better explainability with application in bankruptcy prediction
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Embedding Deep Networks into Visual Explanations
• Explaining Deep Neural Networks
• Saliency Methods for Explaining Adversarial Attacks
• Fast Hierarchical Games for Image Explanations
• Explaining Convolutional Neural Networks using Softmax Gradient Layer-wise Relevance Propagation
• AI for Explaining Decisions in Multi-Agent Environments
• When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures
• Coalitional strategies for efficient individual prediction explanation
• Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification
• Predicting drug response of tumors from integrated genomic profiles by deep neural networks
• Global Explanations of Neural Networks: Mapping the Landscape of Predictions
• Interpret Federated Learning with Shapley Values
• Explanation-Guided Training for Cross-Domain Few-Shot Classification
• Realised Volatility Forecasting: Machine Learning via Financial Word Embedding
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions
• DISCOVER: 2-D Multiview Summarization of Optical Coherence Tomography Angiography for Automatic Diabetic Retinopathy Diagnosis
• Evaluating the Correctness of Explainable AI Algorithms for Classification
• Deep Learning in Protein Structural Modeling and Design
• Adversarial Infidelity Learning for Model Interpretation
• Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
• FairCanary: Rapid Continuous Explainable Fairness
• Refining Language Models with Compositional Explanations
• Interpreting Super-Resolution Networks with Local Attribution Maps
• Interpretable Deep Learning under Fire
• On Baselines for Local Feature Attributions
• Certifiably Robust Interpretation in Deep Learning
• Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation
• Mutual Information for Explainable Deep Learning of Multiscale Systems
• ML-LOO: Detecting Adversarial Examples with Feature Attribution
• Deeply Explain CNN via Hierarchical Decomposition
• Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networks
• Understanding Bias in Machine Learning
• Domain Knowledge Aided Explainable Artificial Intelligence for Intrusion Detection and Response
• On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classification
• Weakly-Supervised Action Localization and Action Recognition using Global-Local Attention of 3D CNN
• A Rate-Distortion Framework for Explaining Neural Network Decisions
• Generative causal explanations of black-box classifiers
• Unbox the Black-box for the Medical Explainable AI via Multi-modal and Multi-centre Data Fusion: A Mini-Review, Two Showcases and Beyond
• Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution
• Best of both worlds: local and global explanations with human-understandable concepts
• How to Manipulate CNNs to Make Them Lie: the GradCAM Case
• Latent-CF: A Simple Baseline for Reverse Counterfactual Explanations
• Fooling Partial Dependence via Data Poisoning
• Human Attention in Fine-grained Classification
• Interpretation of multi-label classification models using shapley values
• Benchmarking the Influence of Pre-training on Explanation Performance in MR Image Classification
• Learn to Interpret Atari Agents
• Feature construction using explanations of individual predictions
• Explaining Convolutional Neural Networks through Attribution-Based Input Sampling and Block-Wise Feature Aggregation
• GeCo: Quality Counterfactual Explanations in Real Time
• Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks
• Unveiling The Factors of Aesthetic Preferences with Explainable AI
• Shapley explainability on the data manifold
• Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential Data
• Melody: Generating and Visualizing Machine Learning Model Summary to Understand Data and Classifiers Together
• What is Interpretable? Using Machine Learning to Design Interpretable Decision-Support Systems
• Joint Explainability and Sensitivity-Aware Federated Deep Learning for Transparent 6G RAN Slicing
• VizADS-B: Analyzing Sequences of ADS-B Images Using Explainable Convolutional LSTM Encoder-Decoder to Detect Cyber Attacks
• Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection
• Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
• Towards Explainable Deep Learning for Credit Lending: A Case Study
• Model Explainability in Deep Learning Based Natural Language Processing
• Certification of embedded systems based on Machine Learning: A survey
• Spatio-Temporal Perturbations for Video Attribution
• Bias, Fairness, and Accountability with AI and ML Algorithms
• LioNets: A Neural-Specific Local Interpretation Technique Exploiting Penultimate Layer Information
• Sparse Oblique Decision Trees: A Tool to Understand and Manipulate Neural Net Features
• Explaining in Style: Training a GAN to explain a classifier in StyleSpace
• Pair the Dots: Jointly Examining Training History and Test Stimuli for Model Interpretability
• Explainable Deep Reinforcement Learning for UAV Autonomous Navigation
• ECINN: Efficient Counterfactuals from Invertible Neural Networks
• What went wrong and when? Instance-wise Feature Importance for Time-series Models
• Explaining Deep Neural Networks using Unsupervised Clustering
• The Twin-System Approach as One Generic Solution for XAI: An Overview of ANN-CBR Twins for Explaining Deep Learning
• Fast Real-time Counterfactual Explanations
• A Note about: Local Explanation Methods for Deep Neural Networks lack Sensitivity to Parameter Values
• Multi-modal Machine Learning for Vehicle Rating Predictions Using Image, Text, and Parametric Data
• Rationalizing Predictions by Adversarial Information Calibration
• Interpreting Deep Neural Networks Through Variable Importance
• Understanding Individual Decisions of CNNs via Contrastive Backpropagation
• SUBPLEX: Towards a Better Understanding of Black Box Model Explanations at the Subpopulation Level
• Self-explanatory Deep Salient Object Detection
• A principle feature analysis
• A simple defense against adversarial attacks on heatmap explanations
• Computationally Efficient Feature Significance and Importance for Machine Learning Models
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models
• A Conceptual Framework for Establishing Trust in Real World Intelligent Systems
• Evaluating neural network explanation methods using hybrid documents and morphological agreement
• On the Evaluation of the Plausibility and Faithfulness of Sentiment Analysis Explanations
• What evidence does deep learning model use to classify Skin Lesions?
• i-Align: an interpretable knowledge graph alignment model
• A Baseline for Shapley Values in MLPs: from Missingness to Neutrality
• Incorporating Priors with Feature Attribution on Text Classification
• Order in the Court: Explainable AI Methods Prone to Disagreement
• XRAI: Better Attributions Through Regions
• Interpretation of Deep Temporal Representations by Selective Visualization of Internally Activated Nodes
• Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
• Have We Learned to Explain?: How Interpretability Methods Can Learn to Encode Predictions in their Interpretations
• Semantics for Global and Local Interpretation of Deep Neural Networks
• Identifying Dominant Industrial Sectors in Market States of the S&P 500 Financial Data
• ProtoryNet - Interpretable Text Classification Via Prototype Trajectories
• Spatio-Temporal Momentum: Jointly Learning Time-Series and Cross-Sectional Strategies
• Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps
• The Shapley Taylor Interaction Index
• Human-interpretable model explainability on high-dimensional data
• CAUSE: Learning Granger Causality from Event Sequences using Attribution Methods
• Consistent Feature Selection for Analytic Deep Neural Networks
• Hide-and-Seek: A Template for Explainable AI
• Information-Theoretic Visual Explanation for Black-Box Classifiers
• Robust Models Are More Interpretable Because Attributions Look Normal
• Shapley Value as Principled Metric for Structured Network Pruning
• Attention Mechanism for Multivariate Time Series Recurrent Model Interpretability Applied to the Ironmaking Industry
• Interpretable Graph Capsule Networks for Object Recognition
• Gaussian Mixture Models for Blended Photometric Redshifts
• Developing Future Human-Centered Smart Cities: Critical Analysis of Smart City Security, Interpretability, and Ethical Challenges
• Bridging the Gap: Gaze Events as Interpretable Concepts to Explain Deep Neural Sequence Models
• Whatcha lookin' at? DeepLIFTing BERT's Attention in Question Answering
• Maximally Invariant Data Perturbation as Explanation
• Improving Feature Attribution through Input-specific Network Pruning
• Series Saliency: Temporal Interpretation for Multivariate Time Series Forecasting
• Towards interpreting ML-based automated malware detection models: a survey
• On The Coherence of Quantitative Evaluation of Visual Explanations
• MonoNet: Towards Interpretable Models by Learning Monotonic Features
• Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networks
• Aggregating explanation methods for stable and robust explainability
• Improving Molecular Graph Neural Network Explainability with Orthonormalization and Induced Sparsity
• Identifying Galaxy Cluster Mergers with Deep Neural Networks using Idealized Compton-y and X-ray maps
• In-Distribution Interpretability for Challenging Modalities
• X-SHAP: towards multiplicative explainability of Machine Learning
• Neural Architecture Search for Joint Optimization of Predictive Power and Biological Knowledge
• Explainable Multivariate Time Series Classification: A Deep Neural Network Which Learns To Attend To Important Variables As Well As Informative Time Intervals
• Interpretable Learning-to-Rank with Generalized Additive Models
• The Need for Standardized Explainability
• Learning Deep Attribution Priors Based On Prior Knowledge
• Optimising for Interpretability: Convolutional Dynamic Alignment Networks
• The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
• Distilling Ensemble of Explanations for Weakly-Supervised Pre-Training of Image Segmentation Models
• Modelling urban networks using Variational Autoencoders
• xAI-GAN: Enhancing Generative Adversarial Networks via Explainable AI Systems
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• An Empirical Study of Explainable AI Techniques on Deep Learning Models For Time Series Tasks
• Discretized Integrated Gradients for Explaining Language Models
• Human Understandable Explanation Extraction for Black-box Classification Models Based on Matrix Factorization
• Just in Time: Personal Temporal Insights for Altering Model Decisions
• Convolutional Neural Network Interpretability with General Pattern Theory
• Framing Algorithmic Recourse for Anomaly Detection
• Unsupervised Detection of Distinctive Regions on 3D Shapes
• Towards Robust Explanations for Deep Neural Networks
• Multi-Stage Influence Function
• Towards Interpretable Ensemble Learning for Image-based Malware Detection
• Investigating Saturation Effects in Integrated Gradients
• Towards Human-Interpretable Prototypes for Visual Assessment of Image Classification Models
• Contextual Prediction Difference Analysis for Explaining Individual Image Classifications
• Predicting traffic overflows on private peering
• Knowledge-based XAI through CBR: There is more to explanations than models can tell
• Gradient Weighted Superpixels for Interpretability in CNNs
• Training and Predicting Visual Error for Real-Time Applications
• The Penalty Imposed by Ablated Data Augmentation
• A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution
• Unveiling Black-boxes: Explainable Deep Learning Models for Patent Classification
• Born Identity Network: Multi-way Counterfactual Map Generation to Explain a Classifier's Decision
• A Spatiotemporal Radar-Based Precipitation Model for Water Level Prediction and Flood Forecasting
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• Explaining Natural Language Processing Classifiers with Occlusion and Language Modeling
• Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing
• Quantitative Evaluation of Explainable Graph Neural Networks for Molecular Property Prediction
• GANMEX: One-vs-One Attributions Guided by GAN-based Counterfactual Explanation Baselines
• Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context
• EMAP: Explanation by Minimal Adversarial Perturbation
• Attribution Analysis of Grammatical Dependencies in LSTMs
• Neural Greedy Pursuit for Feature Selection
• WSAM: Visual Explanations from Style Augmentation as Adversarial Attacker and Their Influence in Image Classification
• Towards Visually Explaining Video Understanding Networks with Perturbation
• SCOUT: Self-aware Discriminant Counterfactual Explanations
• Identifying Critical Neurons in ANN Architectures using Mixed Integer Programming
• SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers
• Model Interpretation and Explainability: Towards Creating Transparency in Prediction Models
• A Comparison of Code Embeddings and Beyond
• VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Models
• FastSHAP: Real-Time Shapley Value Estimation
• Towards Interpretable and Transferable Speech Emotion Recognition: Latent Representation Based Analysis of Features, Methods and Corpora
• Model Interpretability and Rationale Extraction by Input Mask Optimization
• Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning
• Designing Counterfactual Generators using Deep Model Inversion
• EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case
• Evaluating Tree Explanation Methods for Anomaly Reasoning: A Case Study of SHAP TreeExplainer and TreeInterpreter
• FIND: Human-in-the-Loop Debugging Deep Text Classifiers
• Leveraging Activation Maximization and Generative Adversarial Training to Recognize and Explain Patterns in Natural Areas in Satellite Imagery
• Improving Interpretability of Deep Neural Networks in Medical Diagnosis by Investigating the Individual Units
• CXR-Net: An Artificial Intelligence Pipeline for Quick Covid-19 Screening of Chest X-Rays
• Explainability Guided Multi-Site COVID-19 CT Classification
• Ontology-based Interpretable Machine Learning for Textual Data
• What Do Adversarially Robust Models Look At?
• Measuring and improving the quality of visual explanations
• Shapley Explanation Networks
• Evaluation of Saliency-based Explainability Method
• An Empirical Study on the Relation between Network Interpretability and Adversarial Robustness
• Switched linear projections for neural network interpretability
• Fine Grained Dataflow Tracking with Proximal Gradients
• Neural Image Compression and Explanation
• Explaining Image Classifiers using Statistical Fault Localization
• Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding
• Understanding and Diagnosing Vulnerability under Adversarial Attacks
• Scalable, Axiomatic Explanations of Deep Alzheimer's Diagnosis from Heterogeneous Data
• MMLSpark: Unifying Machine Learning Ecosystems at Massive Scales
• Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification
• Optimal Piecewise Local-Linear Approximations
• Towards Explanation of DNN-based Prediction with Guided Feature Inversion
• A Step Towards Exposing Bias in Trained Deep Convolutional Neural Network Models
• Toward the Understanding of Deep Text Matching Models for Information Retrieval
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy
• Discriminative Attribution from Counterfactuals
• Visualizing Color-wise Saliency of Black-Box Image Classification Models
• Learning by Active Forgetting for Neural Networks
• Explanations for Occluded Images
• Class Introspection: A Novel Technique for Detecting Unlabeled Subclasses by Leveraging Classifier Explainability Methods
• Explaining GNN over Evolving Graphs using Information Flow
• Explicating feature contribution using Random Forest proximity distances
• BERT-Beta: A Proactive Probabilistic Approach to Text Moderation
• BayesGrad: Explaining Predictions of Graph Convolutional Networks
• A Tour of Convolutional Networks Guided by Linear Interpreters
• Do Input Gradients Highlight Discriminative Features?
• A General Taylor Framework for Unifying and Revisiting Attribution Methods
• Human-in-the-loop model explanation via verbatim boundary identification in generated neighborhoods
• DRR4Covid: Learning Automated COVID-19 Infection Segmentation from Digitally Reconstructed Radiographs
• Improved Feature Importance Computations for Tree Models: Shapley vs. Banzhaf
• Deep interpretability for GWAS
• Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy
• Harnessing value from data science in business: ensuring explainability and fairness of solutions
• The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results
• Exploring layerwise decision making in DNNs
• Evaluation of Local Explanation Methods for Multivariate Time Series Forecasting
• Logic Traps in Evaluating Attribution Scores
• Uncertainty-aware INVASE: Enhanced Breast Cancer Diagnosis Feature Selection
• High resolution weakly supervised localization architectures for medical images
• Skin Deep Unlearning: Artefact and Instrument Debiasing in the Context of Melanoma Classification
• DEPARA: Deep Attribution Graph for Deep Knowledge Transferability
• Explainability-aided Domain Generalization for Image Classification
• Towards Interpretable ANNs: An Exact Transformation to Multi-Class Multivariate Decision Trees
• Understanding Character Recognition using Visual Explanations Derived from the Human Visual System and Deep Networks
• Making Document-Level Information Extraction Right for the Right Reasons
• Perceptual Score: What Data Modalities Does Your Model Perceive?
• Relevance Attack on Detectors
• DANCE: Enhancing saliency maps using decoys
• Interpreting A Pre-trained Model Is A Key For Model Architecture Optimization: A Case Study On Wav2Vec 2.0
• Introspective Learning by Distilling Knowledge from Online Self-explanation
• Human-grounded Evaluations of Explanation Methods for Text Classification
• Efficient Modelling Across Time of Human Actions and Interactions
• Interpretable Summaries of Black Box Incident Triaging with Subgroup Discovery
• Regularizing Explanations in Bayesian Convolutional Neural Networks
• Less is More: Feature Selection for Adversarial Robustness with Compressive Counter-Adversarial Attacks
• An Empirical Study towards Understanding How Deep Convolutional Nets Recognize Falls
• Probabilistic Selective Encryption of Convolutional Neural Networks for Hierarchical Services
• Comparative Study of Language Models on Cross-Domain Data with Model Agnostic Explainability
• Biophysical models of cis-regulation as interpretable neural networks
• Rank Projection Trees for Multilevel Neural Network Interpretation
• Do not explain without context: addressing the blind spot of model explanations
• Information-theoretic Evolution of Model Agnostic Global Explanations
• Local Explanation of Dialogue Response Generation
• SEEN: Sharpening Explanations for Graph Neural Networks using Explanations from Neighborhoods
• Understanding Misclassifications by Attributes
• T3-Vis: a visual analytic framework for Training and fine-Tuning Transformers in NLP
• Unsupervised Detection and Explanation of Latent-class Contextual Anomalies
• Prediction of Hereditary Cancers Using Neural Networks
• Longitudinal Distance: Towards Accountable Instance Attribution
• Explainability Requires Interactivity
• A Methodology for Exploring Deep Convolutional Features in Relation to Hand-Crafted Features with an Application to Music Audio Modeling
• Attribution Mask: Filtering Out Irrelevant Features By Recursively Focusing Attention on Inputs of DNNs
• Human-Understandable Decision Making for Visual Recognition
• CACTUS: Detecting and Resolving Conflicts in Objective Functions
• Leveraging Model Interpretability and Stability to increase Model Robustness
• Diagnosis and Analysis of Celiac Disease and Environmental Enteropathy on Biopsy Images using Deep Learning Approaches
• Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools
• Dividing Deep Learning Model for Continuous Anomaly Detection of Inconsistent ICT Systems
• Regularized Operating Envelope with Interpretability and Implementability Constraints
• XAlgo: a Design Probe of Explaining Algorithms' Internal States via Question-Answering
• Mutual Information Preserving Back-propagation: Learn to Invert for Faithful Attribution
• Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation
• Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models
• Visual Summary of Value-level Feature Attribution in Prediction Classes with Recurrent Neural Networks
• Interpreting Deep Neural Networks with Relative Sectional Propagation by Analyzing Comparative Gradients and Hostile Activations
• Rigorous Explanation of Inference on Probabilistic Graphical Models
• Interpretability of Blackbox Machine Learning Models through Dataview Extraction and Shadow Model creation
• Learning Propagation Rules for Attribution Map Generation
• Marginal Contribution Feature Importance -- an Axiomatic Approach for The Natural Case
• Automated Dependence Plots
• Play Fair: Frame Attributions in Video Models
• A Robust Interpretable Deep Learning Classifier for Heart Anomaly Detection Without Segmentation
• Model-Agnostic Explanations using Minimal Forcing Subsets
• Unsupervised discovery of Interpretable Visual Concepts
• An Investigation of Language Model Interpretability via Sentence Editing
• Saliency strikes back: How filtering out high frequencies improves white-box explanations
• Counterfactual Explanations via Latent Space Projection and Interpolation
• Logic and the $2$-Simplicial Transformer
• Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing
• Explainable Deep Reinforcement Learning for Portfolio Management: An Empirical Approach
• Based on Graph-VAE Model to Predict Student's Score
• AI challenges for predicting the impact of mutations on protein stability
• Towards Self-Explainable Graph Neural Network
• Defense Against Explanation Manipulation
• Coalitional Bayesian Autoencoders -- Towards explainable unsupervised deep learning
• NeuroView: Explainable Deep Network Decision Making
• Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units
• Logic Constraints to Feature Importances
• Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution
• A Framework for Rationale Extraction for Deep QA models
• Counterfactual Graphs for Explainable Classification of Brain Networks
• Improving Attribution Methods by Learning Submodular Functions
• Explaining a prediction in some nonlinear models
• Explainable Deep Modeling of Tabular Data using TableGraphNet
• Balancing Robustness and Sensitivity using Feature Contrastive Learning
• A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines
• Visualizing Classification Structure of Large-Scale Classifiers
• How to Explain Neural Networks: an Approximation Perspective
• Sanity Simulations for Saliency Methods