Visualizing the Feature Importance for Black Box Models

Abstract

In recent years, a large amount of model-agnostic methods to improve the transparency, trustability and interpretability of machine learning models have been developed. We introduce local feature importance as a local version of a recent model-agnostic global feature importance method. Based on local feature importance, we propose two visual tools: partial importance (PI) and individual conditional importance (ICI) plots which visualize how changes in a feature affect the model performance on average, as well as for individual observations. Our proposed methods are related to partial dependence (PD) and individual conditional expectation (ICE) plots, but visualize the expected (conditional) feature importance instead of the expected (conditional) prediction. Furthermore, we show that averaging ICI curves across observations yields a PI curve, and integrating the PI curve with respect to the distribution of the considered feature results in the global feature importance. Another contribution of our paper is the Shapley feature importance, which fairly distributes the overall performance of a model among the features according to the marginal contributions and which can be used to compare the feature importance across different models.

arXiv commentTo Appear in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10 to 14, 2018, Proceedings, Part I

References in corpus (3)

• OpenML: networked science in machine learning
• Correlation and variable importance in random forests
• OpenML: An R Package to Connect to the Machine Learning Platform OpenML

Cited by in corpus (12)

• Interpretable Machine Learning -- A Brief History, State-of-the-Art and Challenges
• A computational framework for physics-informed symbolic regression with straightforward integration of domain knowledge
• Grouped Feature Importance and Combined Features Effect Plot
• Quantifying Model Complexity via Functional Decomposition for Better Post-Hoc Interpretability
• Incremental Permutation Feature Importance (iPFI): Towards Online Explanations on Data Streams
• Sampling, Intervention, Prediction, Aggregation: A Generalized Framework for Model-Agnostic Interpretations
• Scientific Inference With Interpretable Machine Learning: Analyzing Models to Learn About Real-World Phenomena
• Coalitional strategies for efficient individual prediction explanation
• Assessing high-order effects in feature importance via predictability decomposition
• iSAGE: An Incremental Version of SAGE for Online Explanation on Data Streams
• On the Robustness of Global Feature Effect Explanations
• On Machine Learning-Driven Surrogates for Sound Transmission Loss Simulations