Interpreting Deep Neural Networks Through Variable Importance
arXiv:1901.09839
Abstract
While the success of deep neural networks (DNNs) is well-established across a variety of domains, our ability to explain and interpret these methods is limited. Unlike previously proposed local methods which try to explain particular classification decisions, we focus on global interpretability and ask a universally applicable question: given a trained model, which features are the most important? In the context of neural networks, a feature is rarely important on its own, so our strategy is specifically designed to leverage partial covariance structures and incorporate variable dependence into feature ranking. Our methodological contributions in this paper are two-fold. First, we propose an effect size analogue for DNNs that is appropriate for applications with highly collinear predictors (ubiquitous in computer vision). Second, we extend the recently proposed "RelATive cEntrality" (RATE) measure (Crawford et al., 2019) to the Bayesian deep learning setting. RATE applies an information theoretic criterion to the posterior distribution of effect sizes to assess feature significance. We apply our framework to three broad application areas: computer vision, natural language processing, and social science.
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Scikit-learn: Machine Learning in Python
- Distilling the Knowledge in a Neural Network
- Towards A Rigorous Science of Interpretable Machine Learning
- Axiomatic Attribution for Deep Networks
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Learning Important Features Through Propagating Activation Differences
- Do Deep Nets Really Need to be Deep?
- Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
- Weight Uncertainty in Neural Networks
- SmoothGrad: removing noise by adding noise
- A note on the group lasso and a sparse group lasso
- Sanity Checks for Saliency Maps
- Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
- Variational Dropout and the Local Reparameterization Trick
- One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
- Distilling a Neural Network Into a Soft Decision Tree
- Grouped variable importance with random forests and application to multiple functional data analysis
- Proposed Guidelines for the Responsible Use of Explainable Machine Learning