Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models

Abstract

With the availability of large databases and recent improvements in deep learning methodology, the performance of AI systems is reaching or even exceeding the human level on an increasing number of complex tasks. Impressive examples of this development can be found in domains such as image classification, sentiment analysis, speech understanding or strategic game playing. However, because of their nested non-linear structure, these highly successful machine learning and artificial intelligence models are usually applied in a black box manner, i.e., no information is provided about what exactly makes them arrive at their predictions. Since this lack of transparency can be a major drawback, e.g., in medical applications, the development of methods for visualizing, explaining and interpreting deep learning models has recently attracted increasing attention. This paper summarizes recent developments in this field and makes a plea for more interpretability in artificial intelligence. Furthermore, it presents two approaches to explaining predictions of deep learning models, one method which computes the sensitivity of the prediction with respect to changes in the input and one approach which meaningfully decomposes the decision in terms of the input variables. These methods are evaluated on three classification tasks.

8 pages, 2 figures

References in corpus (12)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Towards A Rigorous Science of Interpretable Machine Learning
• Methods for Interpreting and Understanding Deep Neural Networks
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks

Cited by in corpus (132)

• A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Explainable Machine Learning for Scientific Insights and Discoveries
• A Survey on the Explainability of Supervised Machine Learning
• The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies
• Explaining Explanations in AI
• Deep neural network models for computational histopathology: A survey
• What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research
• Deep Neural Networks for Bot Detection
• Deep Learning: A Critical Appraisal
• Model Based Control of Soft Robots: A Survey of the State of the Art and Open Challenges
• Explanations in Autonomous Driving: A Survey
• The Role of Machine Learning in Cybersecurity
• Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction
• Radiological images and machine learning: trends, perspectives, and prospects
• EEG based Major Depressive disorder and Bipolar disorder detection using Neural Networks: A review
• The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
• Trustworthy AI and Robotics and the Implications for the AEC Industry: A Systematic Literature Review and Future Potentials
• Exploring the Role of Visual Content in Fake News Detection
• BaitWatcher: A lightweight web interface for the detection of incongruent news headlines
• Discover Your Social Identity from What You Tweet: a Content Based Approach
• Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
• Study of Different Deep Learning Approach with Explainable AI for Screening Patients with COVID-19 Symptoms: Using CT Scan and Chest X-ray Image Dataset
• Feature selection revisited in the single-cell era
• Explainable Artificial Intelligence Approaches: A Survey
• Explainable Artificial Intelligence: a Systematic Review
• Quantum Algorithms for Deep Convolutional Neural Networks
• On the Explanation of Machine Learning Predictions in Clinical Gait Analysis
• Unsupervised Social Bot Detection via Structural Information Theory
• XAIR: A Framework of Explainable AI in Augmented Reality
• Segment blockchain: A size reduced storage mechanism for blockchain
• Explainable AI, but explainable to whom?
• Decay Replay Mining to Predict Next Process Events
• Ensembles of Convolutional Neural Networks models for pediatric pneumonia diagnosis
• Improving healthcare access management by predicting patient no-show behaviour
• Checklist for responsible deep learning modeling of medical images based on COVID-19 detection studies
• On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors
• Uncertainty-Aware Deep Ensembles for Reliable and Explainable Predictions of Clinical Time Series
• Mediating Community-AI Interaction through Situated Explanation: The Case of AI-Led Moderation
• Building Interpretable Models for Business Process Prediction using Shared and Specialised Attention Mechanisms
• Interpreting a Recurrent Neural Network's Predictions of ICU Mortality Risk
• Machine Learning Security in Industry: A Quantitative Survey
• Responsible AI by Design in Practice
• Deep Neural Networks for Pattern Recognition
• A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
• Explainable Machine Learning with Prior Knowledge: An Overview
• Towards falsifiable interpretability research
• Kandinsky Patterns
• Can counterfactual explanations of AI systems' predictions skew lay users' causal intuitions about the world? If so, can we correct for that?
• Impact of the characteristics of quantum chemical databases on machine learning predictions of tautomerization energies
• Towards Explainable NLP: A Generative Explanation Framework for Text Classification
• Improving the Timing Resolution of Positron Emission Tomography Detectors Using Boosted Learning -- A Residual Physics Approach
• An explainable deep vision system for animal classification and detection in trail-camera images with automatic post-deployment retraining
• Can Explainable AI Explain Unfairness? A Framework for Evaluating Explainable AI
• Explainable artificial intelligence model to predict acute critical illness from electronic health records
• How do Mixture Density RNNs Predict the Future?
• When Explainability Meets Adversarial Learning: Detecting Adversarial Examples using SHAP Signatures
• The Emerging Landscape of Explainable AI Planning and Decision Making
• DeepView: Visualizing Classification Boundaries of Deep Neural Networks as Scatter Plots Using Discriminative Dimensionality Reduction
• Ask to Know More: Generating Counterfactual Explanations for Fake Claims
• Can Requirements Engineering Support Explainable Artificial Intelligence? Towards a User-Centric Approach for Explainability Requirements
• Visualization of AI Systems in Virtual Reality: A Comprehensive Review
• LightGuider: Guiding Interactive Lighting Design using Suggestions, Provenance, and Quality Visualization
• Towards Counterfactual and Contrastive Explainability and Transparency of DCNN Image Classifiers
• Axes for Sociotechnical Inquiry in AI Research
• Explaining Any ML Model? -- On Goals and Capabilities of XAI
• Intensional Artificial Intelligence: From Symbol Emergence to Explainable and Empathetic AI
• A Gradient Mapping Guided Explainable Deep Neural Network for Extracapsular Extension Identification in 3D Head and Neck Cancer Computed Tomography Images
• An Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networks
• AI-Driven Interface Design for Intelligent Tutoring System Improves Student Engagement
• Efficient Training of Robust Decision Trees Against Adversarial Examples
• The TESS Triple-9 Catalog II: a new set of 999 uniformly-vetted exoplanet candidates
• Efficient computation of counterfactual explanations of LVQ models
• DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification
• Deep unfolding of the weighted MMSE beamforming algorithm
• On the Road with 16 Neurons: Mental Imagery with Bio-inspired Deep Neural Networks
• Why an Android App is Classified as Malware? Towards Malware Classification Interpretation
• Pre-gen metrics: Predicting caption quality metrics without generating captions
• Quantifying the amount of visual information used by neural caption generators
• "How Does It Detect A Malicious App?" Explaining the Predictions of AI-based Android Malware Detector
• What Is Considered Complete for Visual Recognition?
• An Embedded Feature Selection Framework for Control
• Layer-wise Relevance Propagation for Explainable Recommendations
• LionForests: Local Interpretation of Random Forests
• Disentangled Neural Relational Inference for Interpretable Motion Prediction
• Towards Efficient Visual Simplification of Computational Graphs in Deep Neural Networks
• Server, server in the cloud. Who is the fairest in the crowd?
• Language is Power: Representing States Using Natural Language in Reinforcement Learning
• AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient
• PhilaeX: Explaining the Failure and Success of AI Models in Malware Detection
• Occluded object reconstruction for first responders with augmented reality glasses using conditional generative adversarial networks
• Founding The Domain of AI Forensics
• Explainable Deep Convolutional Candlestick Learner
• Symbolic Semantic Segmentation and Interpretation of COVID-19 Lung Infections in Chest CT volumes based on Emergent Languages
• Evaluating the performance of the LIME and Grad-CAM explanation methods on a LEGO multi-label image classification task
• Responsible and Representative Multimodal Data Acquisition and Analysis: On Auditability, Benchmarking, Confidence, Data-Reliance & Explainability
• ExplainFix: Explainable Spatially Fixed Deep Networks
• Visual-Semantic Embedding Model Informed by Structured Knowledge
• Fine-grained Uncertainty Modeling in Neural Networks
• Shapley Interpretation and Activation in Neural Networks
• Robot Mindreading and the Problem of Trust
• Explaining Regression Based Neural Network Model
• Architecture for a multilingual Wikipedia
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• You Can Do Better! If You Elaborate the Reason When Making Prediction
• Interpreting Deep Learning based Cerebral Palsy Prediction with Channel Attention
• Deep Quantum Graph Dreaming: Deciphering Neural Network Insights into Quantum Experiments
• Mathematical decisions and non-causal elements of explainable AI
• Position Paper: Towards Transparent Machine Learning
• Survey of Deep Learning Methods for Inverse Problems
• Generalization on the Enhancement of Layerwise Relevance Interpretability of Deep Neural Network
• A Statistician Teaches Deep Learning
• New Perspective of Interpretability of Deep Neural Networks
• Human-grounded Evaluations of Explanation Methods for Text Classification
• Actionable Interpretation of Machine Learning Models for Sequential Data: Dementia-related Agitation Use Case
• An AI-Augmented Lesion Detection Framework For Liver Metastases With Model Interpretability
• Detecting Patterns of Physiological Response to Hemodynamic Stress via Unsupervised Deep Learning
• Investigating bankruptcy prediction models in the presence of extreme class imbalance and multiple stages of economy
• Explaining Predictions of Deep Neural Classifier via Activation Analysis
• Quantum Algorithms for Unsupervised Machine Learning and Neural Networks
• Neurons Activation Visualization and Information Theoretic Analysis
• Identification of state functions by physically-guided neural networks with physically-meaningful internal layers
• Necessary and Sufficient Explanations in Abstract Argumentation
• A2: Extracting Cyclic Switchings from DOB-nets for Rejecting Excessive Disturbances
• DRILL-- Deep Reinforcement Learning for Refinement Operators in $\mathcal{A}\mathcal{LC}$
• A Gaze Data-based Comparative Study to Build a Trustworthy Human-AI Collaboration in Crash Anticipation
• Understanding Information Processing in Human Brain by Interpreting Machine Learning Models
• Maintaining The Humanity of Our Models
• User-Interactive Machine Learning Model for Identifying Structural Relationships of Code Features
• Explainable Fuzzy Utility Mining on Sequences
• A New Framework for Machine Intelligence: Concepts and Prototype
• Computational principles of intelligence: learning and reasoning with neural networks