Explaining Predictive Uncertainty by Exposing Second-Order Effects
arXiv:2401.17441 · doi:10.1016/j.patcog.2024.111171
Abstract
Explainable AI has brought transparency into complex ML blackboxes, enabling, in particular, to identify which features these models use for their predictions. So far, the question of explaining predictive uncertainty, i.e. why a model 'doubts', has been scarcely studied. Our investigation reveals that predictive uncertainty is dominated by second-order effects, involving single features or product interactions between them. We contribute a new method for explaining predictive uncertainty based on these second-order effects. Computationally, our method reduces to a simple covariance computation over a collection of first-order explanations. Our method is generally applicable, allowing for turning common attribution techniques (LRP, Gradient x Input, etc.) into powerful second-order uncertainty explainers, which we call CovLRP, CovGI, etc. The accuracy of the explanations our method produces is demonstrated through systematic quantitative evaluations, and the overall usefulness of our method is demonstrated via two practical showcases.
12 pages + supplement
References in corpus (9)
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
- Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
- Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
- Confidence Calibration and Predictive Uncertainty Estimation for Deep Medical Image Segmentation
- Higher-Order Explanations of Graph Neural Networks via Relevant Walks
- Uncertainty-Aware Reinforcement Learning for Collision Avoidance
- On the Explanation of Machine Learning Predictions in Clinical Gait Analysis
- Building and Interpreting Deep Similarity Models
- Getting a CLUE: A Method for Explaining Uncertainty Estimates