Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications

Abstract

With the broader and highly successful usage of machine learning in industry and the sciences, there has been a growing demand for Explainable AI. Interpretability and explanation methods for gaining a better understanding about the problem solving abilities and strategies of nonlinear Machine Learning, in particular, deep neural networks, are therefore receiving increased attention. In this work we aim to (1) provide a timely overview of this active emerging field, with a focus on 'post-hoc' explanations, and explain its theoretical foundations, (2) put interpretability algorithms to a test both from a theory and comparative evaluation perspective using extensive simulations, (3) outline best practice aspects i.e. how to best include interpretation methods into the standard usage of machine learning and (4) demonstrate successful usage of explainable AI in a representative selection of application scenarios. Finally, we discuss challenges and possible future directions of this exciting foundational field of machine learning.

arXiv comment30 pages, 20 figures

References in corpus (64)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Learning in Neural Networks: An Overview
• Neural Machine Translation by Jointly Learning to Align and Translate
• Explaining and Harnessing Adversarial Examples
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Intriguing properties of neural networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Methods for Interpreting and Understanding Deep Neural Networks
• Striving for Simplicity: The All Convolutional Net
• Learning Important Features Through Propagating Activation Differences
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Explainable Machine Learning for Scientific Insights and Discoveries
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• Interpretable Explanations of Black Boxes by Meaningful Perturbation
• A Unifying Review of Deep and Shallow Anomaly Detection
• On the Number of Linear Regions of Deep Neural Networks
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• GNNExplainer: Generating Explanations for Graph Neural Networks
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Deep image mining for diabetic retinopathy screening
• Actionable Recourse in Linear Classification
• Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Understanding the Role of Individual Units in a Deep Neural Network
• Deep neural networks can predict mortality from 12-lead electrocardiogram voltage data
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Learning how to explain neural networks: PatternNet and PatternAttribution
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• Higher-Order Explanations of Graph Neural Networks via Relevant Walks
• Explaining the Unique Nature of Individual Gait Patterns with Deep Learning
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• Local Rule-Based Explanations of Black Box Decision Systems
• Resolving challenges in deep learning-based analyses of histopathological images using explanation methods
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Explanations can be manipulated and geometry is to blame
• NeuralHydrology -- Interpreting LSTMs in Hydrology
• Galaxy Morphology Classification with Deep Convolutional Neural Networks
• Fooling Neural Network Interpretations via Adversarial Model Manipulation
• From Clustering to Cluster Explanations via Neural Networks
• RUDDER: Return Decomposition for Delayed Rewards
• Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
• Visualizing Convolutional Neural Network Protein-Ligand Scoring
• How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Deep neural network or dermatologist?
• Explaining and Interpreting LSTMs
• How Important Is a Neuron?
• Building and Interpreting Deep Similarity Models
• Graying the black box: Understanding DQNs
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• Local Explanation Methods for Deep Neural Networks Lack Sensitivity to Parameter Values
• Learning Global Pairwise Interactions with Bayesian Neural Networks
• A Rate-Distortion Framework for Explaining Neural Network Decisions
• Computationally Efficient Measures of Internal Neuron Importance
• Contextual Prediction Difference Analysis for Explaining Individual Image Classifications
• Finding and Removing Clever Hans: Using Explanation Methods to Debug and Improve Deep Models

Cited by in corpus (88)

• Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• SpookyNet: Learning Force Fields with Electronic Degrees of Freedom and Nonlocal Effects
• Higher-Order Explanations of Graph Neural Networks via Relevant Walks
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
• Evaluating explainable artificial intelligence methods for multi-label deep learning classification tasks in remote sensing
• Explainable artificial intelligence in breast cancer detection and risk prediction: A systematic scoping review
• From Clustering to Cluster Explanations via Neural Networks
• Toward Explainable AI for Regression Models
• From Modern CNNs to Vision Transformers: Assessing the Performance, Robustness, and Classification Strategies of Deep Learning Models in Histopathology
• On the Explanation of Machine Learning Predictions in Clinical Gait Analysis
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• Reflective Artificial Intelligence
• Explainable Artificial Intelligence for Bayesian Neural Networks: Towards trustworthy predictions of ocean dynamics
• On Tackling Explanation Redundancy in Decision Trees
• Characterization of anomalous diffusion classical statistics powered by deep learning (CONDOR)
• Analysis of a Deep Learning Model for 12-Lead ECG Classification Reveals Learned Features Similar to Diagnostic Criteria
• An XAI framework for robust and transparent data-driven wind turbine power curve models
• Explaining Deep Learning for ECG Analysis: Building Blocks for Auditing and Knowledge Discovery
• The Clever Hans Effect in Unsupervised Learning
• Ripple Knowledge Graph Convolutional Networks For Recommendation Systems
• Hypothesis Testing and Machine Learning: Interpreting Variable Effects in Deep Artificial Neural Networks using Cohen's f2
• When and How to Fool Explainable Models (and Humans) with Adversarial Examples
• Insights Into the Inner Workings of Transformer Models for Protein Function Prediction
• 3D microstructural generation from 2D images of cement paste using generative adversarial networks
• Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
• Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models
• Analyzing Atomic Interactions in Molecules as Learned by Neural Networks
• Evolutionary approaches to explainable machine learning
• Evaluating Visual Explanations of Attention Maps for Transformer-based Medical Imaging
• Explaining Bayesian Neural Networks
• Feature Perturbation Augmentation for Reliable Evaluation of Importance Estimators in Neural Networks
• Explainable AI guided unsupervised fault diagnostics for high-voltage circuit breakers
• Explaining Predictive Uncertainty by Exposing Second-Order Effects
• From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models
• PredDiff: Explanations and Interactions from Conditional Expectations
• SoK: Dataset Copyright Auditing in Machine Learning Systems
• Towards consistency of rule-based explainer and black box model -- fusion of rule induction and XAI-based feature importance
• Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Features
• TAME: Attention Mechanism Based Feature Fusion for Generating Explanation Maps of Convolutional Neural Networks
• T-TAME: Trainable Attention Mechanism for Explaining Convolutional Networks and Vision Transformers
• A Comprehensive Survey on Self-Interpretable Neural Networks
• NeuralCMS: A deep learning approach to study Jupiter's interior
• Insightful analysis of historical sources at scales beyond human capabilities using unsupervised Machine Learning and XAI
• Delivering Inflated Explanations
• Learning Domain Invariant Representations by Joint Wasserstein Distance Minimization
• ECQ${}^{\text{x}}$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs
• On the Complexity of SHAP-Score-Based Explanations: Tractability via Knowledge Compilation and Non-Approximability Results
• Human-Centered Evaluation of XAI Methods
• Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
• Log-Gaussian Gamma Processes for Training Bayesian Neural Networks in Raman and CARS Spectroscopies
• "Guess what I'm doing": Extending legibility to sequential decision tasks
• Explanations Based on Item Response Theory (eXirt): A Model-Specific Method to Explain Tree-Ensemble Model in Trust Perspective
• Fragility, Robustness and Antifragility in Deep Learning
• Solving the enigma: Enhancing faithfulness and comprehensibility in explanations of deep networks
• Harmonizing Feature Attributions Across Deep Learning Architectures: Enhancing Interpretability and Consistency
• The Explanation Necessity for Healthcare AI
• Challenges for cognitive decoding using deep learning methods
• Investigating Data Hierarchies in Multifidelity Machine Learning for Excitation Energies
• Explainability in AI Based Applications: A Framework for Comparing Different Techniques
• When Can You Trust Your Explanations? A Robustness Analysis on Feature Importances
• The Impact of Feature Representation on the Accuracy of Photonic Neural Networks
• HUDD: A tool to debug DNNs for safety analysis
• Deep Learning Methods for Daily Wildfire Danger Forecasting
• Enhancing Brain Source Reconstruction by Initializing 3D Neural Networks with Physical Inverse Solutions
• Study of Distractors in Neural Models of Code
• Uncertainties of Satellite-based Essential Climate Variables from Deep Learning
• MMformer with Adaptive Transferable Attention: Advancing Multivariate Time Series Forecasting for Environmental Applications
• Fast and Accurate Explanations of Distance-Based Classifiers by Uncovering Latent Explanatory Structures
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy
• Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation
• Discriminative Attribution from Counterfactuals
• Towards Early Prediction of Human iPSC Reprogramming Success
• A novel framework employing deep multi-attention channels network for the autonomous detection of metastasizing cells through fluorescence microscopy
• Towards Desiderata-Driven Design of Visual Counterfactual Explainers
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Pull-back Geometry of Persistent Homology Encodings
• Interpretability of deep-learning methods applied to large-scale structure surveys
• An explainable three dimension framework to uncover learning patterns: A unified look in variable sulci recognition
• From Street Form to Spatial Justice: Explaining Urban Exercise Inequality via a Triadic SHAP-Informed Framework
• Quantum Framework for Simulating Linear PDEs with Robin Boundary Conditions
• How to Explain Neural Networks: an Approximation Perspective
• Optimal Sampling Density for Nonparametric Regression
• Machine Learning in Epidemiology
• Explain and Improve: LRP-Inference Fine-Tuning for Image Captioning Models