Shapley explainability on the data manifold

Abstract

Explainability in AI is crucial for model development, compliance with regulation, and providing operational nuance to predictions. The Shapley framework for explainability attributes a model's predictions to its input features in a mathematically principled and model-agnostic way. However, general implementations of Shapley explainability make an untenable assumption: that the model's features are uncorrelated. In this work, we demonstrate unambiguous drawbacks of this assumption and develop two solutions to Shapley explainability that respect the data manifold. One solution, based on generative modelling, provides flexible access to data imputations; the other directly learns the Shapley value-function, providing performance and stability at the cost of flexibility. While "off-manifold" Shapley values can (i) give rise to incorrect explanations, (ii) hide implicit model dependence on sensitive attributes, and (iii) lead to unintelligible explanations in higher-dimensional data, on-manifold explainability overcomes these problems.

To appear in ICLR 2021; 9 pages, 6 figures, 2 appendices

References in corpus (4)

• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• True to the Model or True to the Data?
• Fairwashing Explanations with Off-Manifold Detergent
• A Universal Marginalizer for Amortized Inference in Generative Models

Cited by in corpus (7)

• Fast TreeSHAP: Accelerating SHAP Value Computation for Trees
• Shapley Flow: A Graph-based Approach to Interpreting Model Predictions
• Explainability for fair machine learning
• Feature Removal Is a Unifying Principle for Model Explanation Methods
• Human-interpretable model explainability on high-dimensional data
• groupShapley: Efficient prediction explanation with Shapley values for feature groups
• Ensembles of Random SHAPs