Hierarchical interpretations for neural network predictions

Abstract

Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method, agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. Using examples from Stanford Sentiment Treebank and ImageNet, we show that ACD is effective at diagnosing incorrect predictions and identifying dataset bias. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

arXiv commentPublished in ICLR 2019

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Survey on Deep Learning in Medical Image Analysis
• Axiomatic Attribution for Deep Networks
• Striving for Simplicity: The All Convolutional Net
• Interpretable machine learning: definitions, methods, and applications
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Understanding Neural Networks Through Deep Visualization
• Understanding Black-box Predictions via Influence Functions
• SmoothGrad: removing noise by adding noise
• How to Explain Individual Classification Decisions
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Understanding Neural Networks through Representation Erasure
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Distilling a Neural Network Into a Soft Decision Tree
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Real Time Image Saliency for Black Box Classifiers
• Automatic Rule Extraction from Long Short Term Memory Networks
• Interpreting CNN Knowledge via an Explanatory Graph
• Detecting Statistical Interactions from Neural Network Weights
• Interpreting Convolutional Neural Networks Through Compression

Cited by in corpus (30)

• Interpretable machine learning: definitions, methods, and applications
• On Interpretability of Artificial Neural Networks: A Survey
• Explaining Explanations: Axiomatic Feature Interactions for Deep Networks
• A Selective Overview of Deep Learning
• Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking
• Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
• Towards Hierarchical Importance Attribution: Explaining Compositional Semantics for Neural Sequence Models
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Fast Hierarchical Games for Image Explanations
• Can I trust you more? Model-Agnostic Hierarchical Explanations
• A Unified Approach to Interpreting and Boosting Adversarial Transferability
• Disentangled Attribution Curves for Interpreting Random Forests and Boosted Trees
• Sparse Oblique Decision Trees: A Tool to Understand and Manipulate Neural Net Features
• Transformation Importance with Applications to Cosmology
• Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text Classification
• Interpreting and Boosting Dropout from a Game-Theoretic View
• Achievements and Challenges in Explaining Deep Learning based Computer-Aided Diagnosis Systems
• The Shapley Taylor Interaction Index
• Technical Note: Game-Theoretic Interactions of Different Orders
• Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?
• Defining and Quantifying the Emergence of Sparse Concepts in DNNs
• Finding and Fixing Spurious Patterns with Explanations
• Analysis Methods in Neural Language Processing: A Survey
• A Unified Game-Theoretic Interpretation of Adversarial Robustness
• SelfExplain: A Self-Explaining Architecture for Neural Text Classifiers
• Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
• Exclusion and Inclusion -- A model agnostic approach to feature importance in DNNs
• Improving the trustworthiness of image classification models by utilizing bounding-box annotations
• When Stability meets Sufficiency: Informative Explanations that do not Overwhelm
• NeuroView: Explainable Deep Network Decision Making