Shapley variable importance clouds for interpretable machine learning

Abstract

Interpretable machine learning has been focusing on explaining final models that optimize performance. The current state-of-the-art is the Shapley additive explanations (SHAP) that locally explains variable impact on individual predictions, and it is recently extended for a global assessment across the dataset. Recently, Dong and Rudin proposed to extend the investigation to models from the same class as the final model that are "good enough", and identified a previous overclaim of variable importance based on a single model. However, this method does not directly integrate with existing Shapley-based interpretations. We close this gap by proposing a Shapley variable importance cloud that pools information across good models to avoid biased assessments in SHAP analyses of final models, and communicate the findings via novel visualizations. We demonstrate the additional insights gain compared to conventional explanations and Dong and Rudin's method using criminal justice and electronic medical records data.

References in corpus (7)

• Interpretable machine learning: definitions, methods, and applications
• Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) Detection
• Supersparse Linear Integer Models for Optimized Medical Scoring Systems
• CXPlain: Causal Explanations for Model Interpretation under Uncertainty
• Learning Optimized Risk Scores
• Getting a CLUE: A Method for Explaining Uncertainty Estimates
• Feature Removal Is a Unifying Principle for Model Explanation Methods

Cited by in corpus (1)

• A novel interpretable machine learning system to generate clinical risk scores: An application for predicting early mortality or unplanned readmission in a retrospective cohort study