Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
arXiv:2212.14855 · doi:10.1109/TPAMI.2024.3388275
Abstract
Explainable AI aims to overcome the black-box nature of complex ML models like neural networks by generating explanations for their predictions. Explanations often take the form of a heatmap identifying input features (e.g. pixels) that are relevant to the model's decision. These explanations, however, entangle the potentially multiple factors that enter into the overall complex decision strategy. We propose to disentangle explanations by extracting at some intermediate layer of a neural network, subspaces that capture the multiple and distinct activation patterns (e.g. visual concepts) that are relevant to the prediction. To automatically extract these subspaces, we propose two new analyses, extending principles found in PCA or ICA to explanations. These novel analyses, which we call principal relevant component analysis (PRCA) and disentangled relevant subspace analysis (DRSA), maximize relevance instead of e.g. variance or kurtosis. This allows for a much stronger focus of the analysis on what the ML model actually uses for predicting, ignoring activations or concepts to which the model is invariant. Our approach is general enough to work alongside common attribution techniques such as Shapley Value, Integrated Gradients, or LRP. Our proposed methods show to be practically useful and compare favorably to the state of the art as demonstrated on benchmarks and three use cases.
arXiv comment17 pages + supplement
References in corpus (22)
- SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
- Methods for Interpreting and Understanding Deep Neural Networks
- Quantum-Chemical Insights from Deep Tensor Neural Networks
- Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
- Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
- Explainable Machine Learning for Scientific Insights and Discoveries
- Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
- Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
- How to Explain Individual Classification Decisions
- Captum: A unified and generic model interpretability library for PyTorch
- Higher-Order Explanations of Graph Neural Networks via Relevant Walks
- Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
- From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
- Resolving challenges in deep learning-based analyses of histopathological images using explanation methods
- NeuralHydrology -- Interpreting LSTMs in Hydrology
- From Clustering to Cluster Explanations via Neural Networks
- Building and Interpreting Deep Similarity Models
- Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy
- Deeply Explain CNN via Hierarchical Decomposition
- Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees
- Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
- DORA: Exploring Outlier Representations in Deep Neural Networks