A Unified Approach to Interpreting Model Predictions

Abstract

Understanding why a model makes a certain prediction can be as crucial as the prediction's accuracy in many applications. However, the highest accuracy for large modern datasets is often achieved by complex models that even experts struggle to interpret, such as ensemble or deep learning models, creating a tension between accuracy and interpretability. In response, various methods have recently been proposed to help users interpret the predictions of complex models, but it is often unclear how these methods are related and when one method is preferable over another. To address this problem, we present a unified framework for interpreting predictions, SHAP (SHapley Additive exPlanations). SHAP assigns each feature an importance value for a particular prediction. Its novel components include: (1) the identification of a new class of additive feature importance measures, and (2) theoretical results showing there is a unique solution in this class with a set of desirable properties. The new class unifies six existing methods, notable because several recent methods in the class lack the proposed desirable properties. Based on insights from this unification, we present new methods that show improved computational performance and/or better consistency with human intuition than previous approaches.

arXiv commentTo appear in NIPS 2017

References in corpus (2)

• Learning Important Features Through Propagating Activation Differences
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences

Cited by in corpus (274)

• Interpretable machine learning: definitions, methods, and applications
• Time Series Forecasting With Deep Learning: A Survey
• Showing Academic Performance Predictions during Term Planning: Effects on Students' Decisions, Behaviors, and Preferences
• A Survey on the Explainability of Supervised Machine Learning
• Explaining Explanations in AI
• Machine Learning for Integrating Data in Biology and Medicine: Principles, Practice, and Opportunities
• Towards Explainable Artificial Intelligence
• Fairness in Machine Learning: A Survey
• Towards Robust Interpretability with Self-Explaining Neural Networks
• Understanding the Role of Individual Units in a Deep Neural Network
• On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection
• Post-hoc Interpretability for Neural NLP: A Survey
• EnergyStar++: Towards more accurate and explanatory building energy benchmarking
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• A Survey on Kolmogorov-Arnold Network
• Ethical Dimensions of Visualization Research
• Consistent feature attribution for tree ensembles
• XCM: An Explainable Convolutional Neural Network for Multivariate Time Series Classification
• Scientometric Review of Artificial Intelligence for Operations & Maintenance of Wind Turbines: The Past, Present and Future
• Fairway: A Way to Build Fair ML Software
• Machine learning the nuclear mass
• A Prescriptive Learning Analytics Framework: Beyond Predictive Modelling and onto Explainable AI with Prescriptive Analytics and ChatGPT
• Acquisition of Chess Knowledge in AlphaZero
• Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
• Impossibility Theorems for Feature Attribution
• Explanations of model predictions with live and breakDown packages
• State-of-the-Art Methods for Exposure-Health Studies: results from the Exposome Data Challenge Event
• How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations
• Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience
• Deep neural network or dermatologist?
• Utilizing XAI technique to improve autoencoder based model for computer network anomaly detection with shapley additive explanation(SHAP)
• Multi-classifier prediction of knee osteoarthritis progression from incomplete imbalanced longitudinal data
• Logic Explained Networks
• EXPLAIN-IT: Towards Explainable AI for Unsupervised Network Traffic Analysis
• A general framework for inference on algorithm-agnostic variable importance
• Explainability in Music Recommender Systems
• Relationships between distortions of inorganic framework and band gap of layered hybrid halide perovskites
• Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
• Machine learning driven synthesis of few-layered WTe2
• The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
• Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making
• Deep Learning Based Cloud Cover Parameterization for ICON
• Super-App Behavioral Patterns in Credit Risk Models: Financial, Statistical and Regulatory Implications
• Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research
• Training Data Influence Analysis and Estimation: A Survey
• Modelling the dielectric constants of crystals using machine learning
• Explainable Activity Recognition for Smart Home Systems
• Entropy-based Logic Explanations of Neural Networks
• Improvement of a Prediction Model for Heart Failure Survival through Explainable Artificial Intelligence
• Explaining the difference between men's and women's football
• Decoding pedestrian and automated vehicle interactions using immersive virtual reality and interpretable deep learning
• Shapley-based Explainable AI for Clustering Applications in Fault Diagnosis and Prognosis
• Explainable Artificial Intelligence for Pharmacovigilance: What Features Are Important When Predicting Adverse Outcomes?
• Explainability: Relevance based Dynamic Deep Learning Algorithm for Fault Detection and Diagnosis in Chemical Processes
• Explainable AI for Interpretable Credit Scoring
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Multilayer Network Analysis for Improved Credit Risk Prediction
• Towards a data-driven characterization of behavioral changes induced by the seasonal flu
• Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams
• SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localization
• Explainable Artificial Intelligence for Bayesian Neural Networks: Towards trustworthy predictions of ocean dynamics
• On Tackling Explanation Redundancy in Decision Trees
• Visualizing Deep Networks by Optimizing with Integrated Gradients
• Explaining Question Answering Models through Text Generation
• Explain and Predict, and then Predict Again
• AutoPrognosis 2.0: Democratizing Diagnostic and Prognostic Modeling in Healthcare with Automated Machine Learning
• Retention Is All You Need
• Deep Weighted Averaging Classifiers
• Pitfalls and Protocols in Practice of Manufacturing Data Science
• Locally Interpretable Models and Effects based on Supervised Partitioning (LIME-SUP)
• Deep Structured Neural Networks for Turbulence Closure Modelling
• Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud
• Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Making
• Explainable AI for Psychological Profiling from Digital Footprints: A Case Study of Big Five Personality Predictions from Spending Data
• Machine Learning Reveals the State of Intermittent Frictional Dynamics in a Sheared Granular Fault
• Explainable AI Integrated Feature Selection for Landslide Susceptibility Mapping using TreeSHAP
• Can autism be diagnosed with AI?
• The Foundational Capabilities of Large Language Models in Predicting Postoperative Risks Using Clinical Notes
• Tree Space Prototypes: Another Look at Making Tree Ensembles Interpretable
• Sampling, Intervention, Prediction, Aggregation: A Generalized Framework for Model-Agnostic Interpretations
• A Machine Learning Tutorial for Operational Meteorology, Part II: Neural Networks and Deep Learning
• Honest-but-Curious Nets: Sensitive Attributes of Private Inputs Can Be Secretly Coded into the Classifiers' Outputs
• Considerations When Learning Additive Explanations for Black-Box Models
• Interpreting Black Box Models via Hypothesis Testing
• Making Neural Networks Interpretable with Attribution: Application to Implicit Signals Prediction
• A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data
• Epistemic values in feature importance methods: Lessons from feminist epistemology
• A Survey on Neural Network Interpretability
• Granger-causal Attentive Mixtures of Experts: Learning Important Features with Neural Networks
• Evaluation of soccer team defense based on prediction models of ball recovery and being attacked: A pilot study
• Explainable Image Similarity: Integrating Siamese Networks and Grad-CAM
• Model-Based Counterfactual Synthesizer for Interpretation
• Predicting the future success of scientific publications through social network and semantic analysis
• Making Fair ML Software using Trustworthy Explanation
• Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection
• ReLAX: Reinforcement Learning Agent eXplainer for Arbitrary Predictive Models
• When and How to Fool Explainable Models (and Humans) with Adversarial Examples
• Propositionalization and Embeddings: Two Sides of the Same Coin
• Neural Networks for Extreme Quantile Regression with an Application to Forecasting of Flood Risk
• The Bouncer Problem: Challenges to Remote Explainability
• Understanding transit ridership in an equity context through a comparison of statistical and machine learning algorithms
• The Shapley Value of Tuples in Query Answering
• A framework for step-wise explaining how to solve constraint satisfaction problems
• Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation
• The Shapley Value in Database Management
• Question-Driven Design Process for Explainable AI User Experiences
• Fast Hierarchical Games for Image Explanations
• AutoScore-Imbalance: An interpretable machine learning tool for development of clinical scores with rare events data
• ZELDA: fitting Lyman-alpha line profiles using deep learning
• SLISEMAP: Supervised dimensionality reduction through local explanations
• Coalitional strategies for efficient individual prediction explanation
• Automatic Assessment of Functional Movement Screening Exercises with Deep Learning Architectures
• GNNShap: Scalable and Accurate GNN Explanation using Shapley Values
• Roots of Trumpism: Homophily and Social Feedback in Donald Trump Support on Reddit
• Prediction of Blood Lactate Values in Critically Ill Patients: A Retrospective Multi-center Cohort Study
• Machine learning applied to simulations of collisions between rotating, differentiated planets
• Fidelity and Privacy of Synthetic Medical Data
• Calibrated Explanations: with Uncertainty Information and Counterfactuals
• Representation Learning for Electronic Health Records
• Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
• Using Interaction Data to Predict Engagement with Interactive Media
• Instance-wise or Class-wise? A Tale of Neighbor Shapley for Concept-based Explanation
• Adversarial Infidelity Learning for Model Interpretation
• The Shapley Value of Inconsistency Measures for Functional Dependencies
• Timing of the solar wind propagation delay between L1 and Earth based on machine learning
• ESG Rating Disagreement and Corporate Total Factor Productivity:Inference and Prediction
• FairCanary: Rapid Continuous Explainable Fairness
• Efficient learning of large sets of locally optimal classification rules
• Assessment of Neural Network Augmented Reynolds Averaged Navier Stokes Turbulence Model in Extrapolation Modes
• Machine Learning Interpretability of Outer Radiation Belt Enhancement \& Depletion Events
• QoT estimation using EGN-assisted machine learning for multi-period network planning
• Quantitative Evaluations on Saliency Methods: An Experimental Study
• AI Data Readiness Inspector (AIDRIN) for Quantitative Assessment of Data Readiness for AI
• Interpretation of multi-label classification models using shapley values
• Trustworthy AI in practice: an analysis of practitioners' needs and challenges
• SHAPFUZZ: Efficient Fuzzing via Shapley-Guided Byte Selection
• Enhancing operations management through smart sensors: measuring and improving well-being, interaction and performance of logistics workers
• Explaining a Deep Reinforcement Learning Docking Agent Using Linear Model Trees with User Adapted Visualization
• SOLD: Sinhala Offensive Language Dataset
• Comparative and Interpretative Analysis of CNN and Transformer Models in Predicting Wildfire Spread Using Remote Sensing Data
• How Costly is Noise? Data and Disparities in Consumer Credit
• Machine Learning in Biomechanics: Key Applications and Limitations in Walking, Running, and Sports Movements
• Unveiling The Factors of Aesthetic Preferences with Explainable AI
• Efficient Learning of Interpretable Classification Rules
• E-Commerce Dispute Resolution Prediction
• Efficiently Explaining CSPs with Unsatisfiable Subset Optimization (extended algorithms and examples)
• CEST-KAN: Kolmogorov-Arnold Networks for CEST MRI Data Analysis
• AHMoSe: A Knowledge-Based Visual Support System for Selecting Regression Machine Learning Models
• Your Face Mirrors Your Deepest Beliefs-Predicting Personality and Morals through Facial Emotion Recognition
• Local Post-Hoc Explanations for Predictive Process Monitoring in Manufacturing
• Assessment of the influence of features on a classification problem: an application to COVID-19 patients
• A Look into How Machine Learning is Reshaping Engineering Models: the Rise of Analysis Paralysis, Optimal yet Infeasible Solutions, and the Inevitable Rashomon Paradox
• Progressive Disclosure: Designing for Effective Transparency
• iScore: Visual Analytics for Interpreting How Language Models Automatically Score Summaries
• Integrating Wearable Sensor Data and Self-reported Diaries for Personalized Affect Forecasting
• LLpowershap: Logistic Loss-based Automated Shapley Values Feature Selection Method
• In Search of netUnicorn: A Data-Collection Platform to Develop Generalizable ML Models for Network Security Problems
• Impact of Accuracy on Model Interpretations
• Machine learning and natural language processing models to predict the extent of food processing
• Enhancing Psychologists' Understanding through Explainable Deep Learning Framework for ADHD Diagnosis
• Delivering Inflated Explanations
• The Epistemological Consequences of Large Language Models: Rethinking collective intelligence and institutional knowledge
• Augmenting the availability of historical GDP per capita estimates through machine learning
• ASTERYX : A model-Agnostic SaT-basEd appRoach for sYmbolic and score-based eXplanations
• The Impact of COVID-19 Pandemic on Ridesourcing Services Differed Between Small Towns and Large Cities
• Causally motivated Shortcut Removal Using Auxiliary Labels
• A Baseline for Shapley Values in MLPs: from Missingness to Neutrality
• Revealing the Compositional Control of Electrical, Mechanical, Optical, and Physical Properties of Inorganic Glasses
• Synthesizing Action Sequences for Modifying Model Decisions
• SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction
• Efficient Search for Diverse Coherent Explanations
• Exploring the Determinants of Pedestrian Crash Severity Using an AutoML Approach
• PADTHAI-MM: Principles-based Approach for Designing Trustworthy, Human-centered AI using MAST Methodology
• Anomaly Attribution with Likelihood Compensation
• Peptide Classification from Statistical Analysis of Nanopore Translocation Experiments
• Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Models
• Towards Robust Metrics for Concept Representation Evaluation
• An exact counterfactual-example-based approach to tree-ensemble models interpretability
• ODTE -- An ensemble of multi-class SVM-based oblique decision trees
• DeepSeer: Interactive RNN Explanation and Debugging via State Abstraction
• SQLFlow: A Bridge between SQL and Machine Learning
• ControlBurn: Feature Selection by Sparse Forests
• Interpretable Disease Prediction based on Reinforcement Path Reasoning over Knowledge Graphs
• Clustering patterns connecting COVID-19 dynamics and Human mobility using optimal transport
• Towards Explainable Meta-Learning for DDoS Detection
• Counterfactual Explanations in Sequential Decision Making Under Uncertainty
• Information Theoretic Measures for Fairness-aware Feature Selection
• A Comparison Between Tsetlin Machines and Deep Neural Networks in the Context of Recommendation Systems
• Data-Centric Engineering: integrating simulation, machine learning and statistics. Challenges and Opportunities
• Unveiling interatomic distances influencing the reaction coordinates in alanine dipeptide isomerization: An explainable deep learning approach
• On The Coherence of Quantitative Evaluation of Visual Explanations
• Experimental exploration of ErB${}_2$ and SHAP analysis on a machine-learned model of magnetocaloric materials for materials design
• Shapley Value-driven Data Pruning for Recommender Systems
• Polynomial Ridge Flowfield Estimation
• Mining the Stars: Learning Quality Ratings with User-facing Explanations for Vacation Rentals
• The Yield Curve as a Recession Leading Indicator. An Application for Gradient Boosting and Random Forest
• Challenging common interpretability assumptions in feature attribution explanations
• NLPGuard: A Framework for Mitigating the Use of Protected Attributes by NLP Classifiers
• Knowledge-based XAI through CBR: There is more to explanations than models can tell
• A Review of Explainable Artificial Intelligence in Manufacturing
• Analyzing Fairness of Computer Vision and Natural Language Processing Models
• Adversarial Robustness with Non-uniform Perturbations
• Using Machine Learning techniques in phenomenological studies in flavour physics
• Machine learning based stellar classification with highly sparse photometry data
• Evaluating and Characterizing Human Rationales
• Russo-Ukrainian War: Prediction and explanation of Twitter suspension
• Generating Post-hoc Explanations for Skip-gram-based Node Embeddings by Identifying Important Nodes with Bridgeness
• Privacy Inference Attacks and Defenses in Cloud-based Deep Neural Network: A Survey
• Investigating the hyperparameter space of deep neural network models for reaction coordinates
• CAMP: A Context-Aware Cricket Players Performance Metric
• Robust covariance estimation and explainable outlier detection for matrix-valued data
• Site-specific Deterministic Temperature and Humidity Forecasts with Explainable and Reliable Machine Learning
• EnTri: Ensemble Learning with Tri-level Representations for Explainable Scene Recognition
• Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces
• ACTA: A Mobile-Health Solution for Integrated Nudge-Neurofeedback Training for Senior Citizens
• Mask of truth: model sensitivity to unexpected regions of medical images
• Understanding Cycling Mobility: Bologna Case Study
• WATCH: A Workflow to Assess Treatment Effect Heterogeneity in Drug Development for Clinical Trial Sponsors
• Measuring the Driving Forces of Predictive Performance: Application to Credit Scoring
• Seven challenges for harmonizing explainability requirements
• On the Tractability of SHAP Explanations
• Enabling Machine Learning Algorithms for Credit Scoring -- Explainable Artificial Intelligence (XAI) methods for clear understanding complex predictive models
• Progress in deep Markov State Modeling: Coarse graining and experimental data restraints
• Factor Importance Ranking and Selection using Total Indices
• Model Interpretation and Explainability: Towards Creating Transparency in Prediction Models
• OTCliM: generating a near-surface climatology of optical turbulence strength ($C_n^2$) using gradient boosting
• Deep Learning: a Heuristic Three-stage Mechanism for Grid Searches to Optimize the Future Risk Prediction of Breast Cancer Metastasis Using EHR-based Clinical Data
• From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing
• Data-Adaptive Discriminative Feature Localization with Statistically Guaranteed Interpretation
• The Consequences of the Framing of Machine Learning Risk Prediction Models: Evaluation of Sepsis in General Wards
• Unified Shapley Framework to Explain Prediction Drift
• SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection
• On the Performance of Cyber-Biomedical Features for Intrusion Detection in Healthcare 5.0
• SHARQ: Explainability Framework for Association Rules on Relational Data
• Towards Feasible Counterfactual Explanations: A Taxonomy Guided Template-based NLG Method
• Decomposition of Global Feature Importance into Direct and Associative Components (DEDACT)
• What drives bitcoin? An approach from continuous local transfer entropy and deep learning classification models
• Using Slisemap to interpret physical data
• An adversarially robust data-market for spatial, crowd-sourced data
• "I had a solid theory before but it's falling apart": Polarizing Effects of Algorithmic Transparency
• Online Product Feature Recommendations with Interpretable Machine Learning
• Ownership at Large -- Open Problems and Challenges in Ownership Management
• Assessing Social Determinants-Related Performance Bias of Machine Learning Models: A case of Hyperchloremia Prediction in ICU Population
• Predicting Tropical Cyclone Track Forecast Errors using a Probabilistic Neural Network
• GCN-SE: Attention as Explainability for Node Classification in Dynamic Graphs
• A Simple and Interpretable Predictive Model for Healthcare
• Efficient Decompositional Rule Extraction for Deep Neural Networks
• An Earth-System-Oriented View of the S2S Predictability of North American Weather Regimes
• Interpretabilité des modèles : état des lieux des méthodes et application à l'assurance
• Tree in Tree: from Decision Trees to Decision Graphs
• Granularity at Scale: Estimating Neighborhood Socioeconomic Indicators from High-Resolution Orthographic Imagery and Hybrid Learning
• A mapping-free NLP-based technique for sequence search in Nanopore long-reads
• Independent Ethical Assessment of Text Classification Models: A Hate Speech Detection Case Study
• Identifying the post-pandemic determinants of low performing students in Latin America through Interpretable Machine Learning methods
• Machine Learning Algorithms for Financial Asset Price Forecasting
• Learning from Discriminatory Training Data
• Harnessing value from data science in business: ensuring explainability and fairness of solutions
• Prediction can be safely used as a proxy for explanation in causally consistent Bayesian generalized linear models
• Towards Interpretable Multi-Task Learning Using Bilevel Programming
• Impact of Interventional Policies Including Vaccine on Covid-19 Propagation and Socio-Economic Factors
• ExCode-Mixed: Explainable Approaches towards Sentiment Analysis on Code-Mixed Data using BERT models
• Towards Learning a Vocabulary of Visual Concepts and Operators using Deep Neural Networks
• Bridging Breiman's Brook: From Algorithmic Modeling to Statistical Learning
• A Kernel Framework to Quantify a Model's Local Predictive Uncertainty under Data Distributional Shifts
• Feature Extraction and Analysis for GPT-Generated Text
• Understanding peacefulness through the world news
• Logic Constraints to Feature Importances
• Analysis and classification of main risk factors causing stroke in Shanxi Province
• Mutual Information Preserving Back-propagation: Learn to Invert for Faithful Attribution
• S2AMP: A High-Coverage Dataset of Scholarly Mentorship Inferred from Publications
• Applications of Machine Learning to Optimizing Polyolefin Manufacturing
• Towards Comparative Physical Interpretation of Spatial Variability Aware Neural Networks: A Summary of Results
• Text Counterfactuals via Latent Optimization and Shapley-Guided Search
• On Predictive Explanation of Data Anomalies