Explaining Explanations: Axiomatic Feature Interactions for Deep Networks

Abstract

Recent work has shown great promise in explaining neural network behavior. In particular, feature attribution methods explain which features were most important to a model's prediction on a given input. However, for many tasks, simply knowing which features were important to a model's prediction may not provide enough insight to understand model behavior. The interactions between features within the model may better help us understand not only the model, but also why certain features are more important than others. In this work, we present Integrated Hessians, an extension of Integrated Gradients that explains pairwise feature interactions in neural networks. Integrated Hessians overcomes several theoretical limitations of previous methods to explain interactions, and unlike such previous methods is not limited to a specific architecture or class of neural network. Additionally, we find that our method is faster than existing methods when the number of features is large, and outperforms previous methods on existing quantitative benchmarks. Code available at https://github.com/suinleelab/path_explain

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• A Structured Self-attentive Sentence Embedding
• Convolutional Neural Networks for Sentence Classification
• Searching for Activation Functions
• Sanity Checks for Saliency Maps
• Attention is not Explanation
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Fifty Years of Pulsar Candidate Selection: From simple filters to a new principled real-time classification approach
• Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs
• Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
• Hierarchical interpretations for neural network predictions
• Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms
• MAGIX: Model Agnostic Globally Interpretable Explanations
• Robust Attribution Regularization
• Detecting Statistical Interactions from Neural Network Weights
• Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning Interpretation
• Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference
• A Note about: Local Explanation Methods for Deep Neural Networks lack Sensitivity to Parameter Values
• Incorporating Priors with Feature Attribution on Text Classification
• XRAI: Better Attributions Through Regions
• The Shapley Taylor Interaction Index
• Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification

Cited by in corpus (16)

• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Building and Interpreting Deep Similarity Models
• How does this interaction affect me? Interpretable attribution for feature interactions
• Unwrapping The Black Box of Deep ReLU Networks: Interpretability, Diagnostics, and Simplification
• Interpreting and Boosting Dropout from a Game-Theoretic View
• Technical Note: Game-Theoretic Interactions of Different Orders
• Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?
• The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?
• Learning from learning machines: a new generation of AI technology to meet the needs of science
• Defining and Quantifying the Emergence of Sparse Concepts in DNNs
• Why is "Problems" Predictive of Positive Sentiment? A Case Study of Explaining Unintuitive Features in Sentiment Classification
• A Unified Game-Theoretic Interpretation of Adversarial Robustness
• Interpretable Artificial Intelligence through the Lens of Feature Interaction
• Fine-grained Interpretation and Causation Analysis in Deep NLP Models
• Axiomatic Explanations for Visual Search, Retrieval, and Similarity Learning
• Interpreting Attributions and Interactions of Adversarial Attacks