Consistent Individualized Feature Attribution for Tree Ensembles

Abstract

Interpreting predictions from tree ensemble methods such as gradient boosting machines and random forests is important, yet feature attribution for trees is often heuristic and not individualized for each prediction. Here we show that popular feature attribution methods are inconsistent, meaning they can lower a feature's assigned importance when the true impact of that feature actually increases. This is a fundamental problem that casts doubt on any comparison between features. To address it we turn to recent applications of game theory and develop fast exact tree solutions for SHAP (SHapley Additive exPlanation) values, which are the unique consistent and locally accurate attribution values. We then extend SHAP values to interaction effects and define SHAP interaction values. We propose a rich visualization of individualized feature attributions that improves over classic attribution summaries and partial dependence plots, and a unique "supervised" clustering (clustering based on feature attributions). We demonstrate better agreement with human intuition through a user study, exponential improvements in run time, improved clustering performance, and better identification of influential features. An implementation of our algorithm has also been merged into XGBoost and LightGBM, see http://github.com/slundberg/shap for details.

Follow-up to 2017 ICML Workshop arXiv:1706.06060

References in corpus (2)

• Scikit-learn: Machine Learning in Python
• How to Explain Individual Classification Decisions

Cited by in corpus (110)

• Interpretable machine learning: definitions, methods, and applications
• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
• Scalable and Generalizable Social Bot Detection through Data Selection
• EnergyStar++: Towards more accurate and explanatory building energy benchmarking
• Feature relevance quantification in explainable AI: A causal problem
• Problems with Shapley-value-based explanations as feature importance measures
• Explaining Anomalies Detected by Autoencoders Using SHAP
• A Debiased MDI Feature Importance Measure for Random Forests
• Multi-classifier prediction of knee osteoarthritis progression from incomplete imbalanced longitudinal data
• Explainable AI: current status and future directions
• Explainable Artificial Intelligence for Process Mining: A General Overview and Application of a Novel Local Explanation Approach for Predictive Process Monitoring
• Grouped Feature Importance and Combined Features Effect Plot
• Explainable Machine Learning for Public Policy: Use Cases, Gaps, and Research Directions
• Transferable and extensible machine learning derived atomic charges for modeling hybrid nanoporous materials
• Towards a data-driven characterization of behavioral changes induced by the seasonal flu
• Machine Learning Reveals the Seismic Signature of Eruptive Behavior at Piton de la Fournaise Volcano
• Fast TreeSHAP: Accelerating SHAP Value Computation for Trees
• Interpreting a Recurrent Neural Network's Predictions of ICU Mortality Risk
• Explainable AI Integrated Feature Selection for Landslide Susceptibility Mapping using TreeSHAP
• Do Not Trust Additive Explanations
• Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability
• The many Shapley values for model explanation
• How does this interaction affect me? Interpretable attribution for feature interactions
• Machine Learning Estimation of Heterogeneous Treatment Effects with Instruments
• Interpreted machine learning in fluid dynamics: Explaining relaminarisation events in wall-bounded shear flows
• On the Art and Science of Machine Learning Explanations
• Shapley Flow: A Graph-based Approach to Interpreting Model Predictions
• Secondary control activation analysed and predicted with explainable AI
• SHAP values for Explaining CNN-based Text Classification Models
• A Human-Grounded Evaluation of SHAP for Alert Processing
• Exploring deterministic frequency deviations with explainable AI
• Learning Global Pairwise Interactions with Bayesian Neural Networks
• Evaluating the Correctness of Explainable AI Algorithms for Classification
• Machine learning applied to simulations of collisions between rotating, differentiated planets
• Shapley Decomposition of R-Squared in Machine Learning Models
• On Baselines for Local Feature Attributions
• SHAFF: Fast and consistent SHApley eFfect estimates via random Forests
• Discovering and Explaining the Representation Bottleneck of DNNs
• XGBoostLSS -- An extension of XGBoost to probabilistic forecasting
• Clumpiness: Time-domain classification of red-giant evolutionary states
• Can I trust you more? Model-Agnostic Hierarchical Explanations
• Proposed Guidelines for the Responsible Use of Explainable Machine Learning
• Interpretable Anomaly Detection with DIFFI: Depth-based Isolation Forest Feature Importance
• Shapley explainability on the data manifold
• Wasserstein-based fairness interpretability framework for machine learning models
• Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection
• Local Post-Hoc Explanations for Predictive Process Monitoring in Manufacturing
• Disentangled Attribution Curves for Interpreting Random Forests and Boosted Trees
• VINE: Visualizing Statistical Interactions in Black Box Models
• Bias, Fairness, and Accountability with AI and ML Algorithms
• Impact of Accuracy on Model Interpretations
• Generalized Integrated Gradients: A practical method for explaining diverse ensembles
• Combat COVID-19 Infodemic Using Explainable Natural Language Processing Models
• Incorporating Priors with Feature Attribution on Text Classification
• Who's responsible? Jointly quantifying the contribution of the learning algorithm and training data
• Feature Interactions in XGBoost
• Predictive Analytics for Water Asset Management: Machine Learning and Survival Analysis
• Evolutionary model discovery of causal factors behind the socio-agricultural behavior of the ancestral Pueblo
• Technical Note: Game-Theoretic Interactions of Different Orders
• Interpreting and Boosting Dropout from a Game-Theoretic View
• The Shapley Taylor Interaction Index
• PSD2 Explainable AI Model for Credit Scoring
• Case-Based Reasoning for Assisting Domain Experts in Processing Fraud Alerts of Black-Box Machine Learning Models
• Predicting Driver Fatigue in Automated Driving with Explainability
• High Dimensional Model Explanations: an Axiomatic Approach
• Can We Faithfully Represent Masked States to Compute Shapley Values on a DNN?
• The Need for Standardized Explainability
• Modeling Dispositional and Initial learned Trust in Automated Vehicles with Predictability and Explainability
• Self-interpretable Convolutional Neural Networks for Text Classification
• Revealing drivers and risks for power grid frequency stability with explainable AI
• Defining and Quantifying the Emergence of Sparse Concepts in DNNs
• The varying importance of extrinsic factors in the success of startup fundraising: competition at early-stage and networks at growth-stage
• Connecting Interpretability and Robustness in Decision Trees through Separation
• EMAP: Explanation by Minimal Adversarial Perturbation
• Diabetes Mellitus Forecasting Using Population Health Data in Ontario, Canada
• On the Tractability of SHAP Explanations
• Importance measures derived from random forests: characterisation and extension
• Seven challenges for harmonizing explainability requirements
• Deep Gravity: enhancing mobility flows generation with deep neural networks and geographic information
• TREX: Tree-Ensemble Representer-Point Explanations
• Investigating and Simplifying Masking-based Saliency Methods for Model Interpretability
• Explaining Neural Network Predictions on Sentence Pairs via Learning Word-Group Masks
• Energy-Based Learning for Cooperative Games, with Applications to Valuation Problems in Machine Learning
• A Unified Game-Theoretic Interpretation of Adversarial Robustness
• Machine learning in the social and health sciences
• Feature Synergy, Redundancy, and Independence in Global Model Explanations using SHAP Vector Decomposition
• Inferring feature importance with uncertainties in high-dimensional data
• Induction of Non-Monotonic Rules From Statistical Learning Models Using High-Utility Itemset Mining
• Biased Models Have Biased Explanations
• Interpretable Poverty Mapping using Social Media Data, Satellite Images, and Geospatial Information
• Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning
• Nonparametric Variable Screening with Optimal Decision Stumps
• Improved Feature Importance Computations for Tree Models: Shapley vs. Banzhaf
• Joint learning of interpretation and distillation
• Uncertainty-aware Sensitivity Analysis Using Rényi Divergences
• Evaluating the Values of Sources in Transfer Learning
• A Categorisation of Post-hoc Explanations for Predictive Models
• Computational analysis of laminar structure of the human cortex based on local neuron features
• An Investigation of the Impact of COVID-19 Non-Pharmaceutical Interventions and Economic Support Policies on Foreign Exchange Markets with Explainable AI Techniques
• Clusters in Explanation Space: Inferring disease subtypes from model explanations
• Do not explain without context: addressing the blind spot of model explanations
• Understanding peacefulness through the world news
• Relate and Predict: Structure-Aware Prediction with Jointly Optimized Neural DAG
• Hyperparameter Optimization for Forecasting Stock Returns
• Learning to Ask Medical Questions using Reinforcement Learning
• Using Shapley Values and Variational Autoencoders to Explain Predictive Models with Dependent Mixed Features
• An exploration of the influence of path choice in game-theoretic attribution algorithms
• DNN2LR: Automatic Feature Crossing for Credit Scoring
• WRSE -- a non-parametric weighted-resolution ensemble for predicting individual survival distributions in the ICU
• Working women and caste in India: A study of social disadvantage using feature attribution