Investigating Saturation Effects in Integrated Gradients
arXiv:2010.12697
Abstract
Integrated Gradients has become a popular method for post-hoc model interpretability. De-spite its popularity, the composition and relative impact of different regions of the integral path are not well understood. We explore these effects and find that gradients in saturated regions of this path, where model output changes minimally, contribute disproportionately to the computed attribution. We propose a variant of IntegratedGradients which primarily captures gradients in unsaturated regions and evaluate this method on ImageNet classification networks. We find that this attribution technique shows higher model faithfulness and lower sensitivity to noise com-pared with standard Integrated Gradients. A note-book illustrating our computations and results is available at https://github.com/vivekmig/captum-1/tree/ExpandedIG.
arXiv commentPresented at ICML Workshop on Human Interpretability in Machine Learning (WHI 2020)
References in corpus (8)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Axiomatic Attribution for Deep Networks
- Learning Important Features Through Propagating Activation Differences
- RISE: Randomized Input Sampling for Explanation of Black-box Models
- Gradients of Counterfactuals
- How Important Is a Neuron?
- A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks
- Generalized Integrated Gradients: A practical method for explaining diverse ensembles