Shapley Flow: A Graph-based Approach to Interpreting Model Predictions

Abstract

Many existing approaches for estimating feature importance are problematic because they ignore or hide dependencies among features. A causal graph, which encodes the relationships among input variables, can aid in assigning feature importance. However, current approaches that assign credit to nodes in the causal graph fail to explain the entire graph. In light of these limitations, we propose Shapley Flow, a novel approach to interpreting machine learning models. It considers the entire causal graph, and assigns credit to \textit{edges} instead of treating nodes as the fundamental unit of credit assignment. Shapley Flow is the unique solution to a generalization of the Shapley value axioms to directed acyclic graphs. We demonstrate the benefit of using Shapley Flow to reason about the impact of a model's input on its output. In addition to maintaining insights from existing approaches, Shapley Flow extends the flat, set-based, view prevalent in game theory based explanation methods to a deeper, \textit{graph-based}, view. This graph-based view enables users to understand the flow of importance through a system, and reason about potential interventions.

arXiv commentcamera ready version for AISTATS 2021

References in corpus (13)

• Axiomatic Attribution for Deep Networks
• Striving for Simplicity: The All Convolutional Net
• Learning Important Features Through Propagating Activation Differences
• All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously
• Explaining Explanations in AI
• How to Explain Individual Classification Decisions
• Consistent Individualized Feature Attribution for Tree Ensembles
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Feature relevance quantification in explainable AI: A causal problem
• How Important Is a Neuron?
• True to the Model or True to the Data?
• Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability
• Shapley explainability on the data manifold

Cited by in corpus (5)

• Counterfactual Shapley Additive Explanations
• Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning
• Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena
• Improvement-Focused Causal Recourse (ICR)
• Causal Forecasting:Generalization Bounds for Autoregressive Models