Do not explain without context: addressing the blind spot of model explanations

Abstract

The increasing number of regulations and expectations of predictive machine learning models, such as so called right to explanation, has led to a large number of methods promising greater interpretability. High demand has led to a widespread adoption of XAI techniques like Shapley values, Partial Dependence profiles or permutational variable importance. However, we still do not know enough about their properties and how they manifest in the context in which explanations are created by analysts, reviewed by auditors, and interpreted by various stakeholders. This paper highlights a blind spot which, although critical, is often overlooked when monitoring and auditing machine learning models: the effect of the reference data on the explanation calculation. We discuss that many model explanations depend directly or indirectly on the choice of the referenced data distribution. We showcase examples where small changes in the distribution lead to drastic changes in the explanations, such as a change in trend or, alarmingly, a conclusion. Consequently, we postulate that obtaining robust and useful explanations always requires supporting them with a broader context.

References in corpus (9)

• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Sanity Checks for Saliency Maps
• Consistent Individualized Feature Attribution for Tree Ensembles
• Explaining by Removing: A Unified Framework for Model Explanation
• Problems with Shapley-value-based explanations as feature importance measures
• Machine Learning Explainability for External Stakeholders
• The Grammar of Interactive Explanatory Model Analysis
• Fooling Partial Dependence via Data Poisoning