A general framework for inference on algorithm-agnostic variable importance

Abstract

In many applications, it is of interest to assess the relative contribution of features (or subsets of features) toward the goal of predicting a response -- in other words, to gauge the variable importance of features. Most recent work on variable importance assessment has focused on describing the importance of features within the confines of a given prediction algorithm. However, such assessment does not necessarily characterize the prediction potential of features, and may provide a misleading reflection of the intrinsic value of these features. To address this limitation, we propose a general framework for nonparametric inference on interpretable algorithm-agnostic variable importance. We define variable importance as a population-level contrast between the oracle predictiveness of all available features versus all features except those under consideration. We propose a nonparametric efficient estimation procedure that allows the construction of valid confidence intervals, even when machine learning techniques are used. We also outline a valid strategy for testing the null importance hypothesis. Through simulations, we show that our proposal has good operating characteristics, and we illustrate its use with data from a study of an antibody against HIV-1 infection.

69 total pages (35 in the main document, 34 supplementary), 23 figures (4 in the main document, 19 supplementary)

References in corpus (7)

• A Unified Approach to Interpreting Model Predictions
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Interpretable machine learning: definitions, methods, and applications
• Variable importance in binary regression trees and forests
• Efficient nonparametric statistical inference on population feature importance using Shapley values
• A machine learning-based approach for estimating and testing associations with multivariate outcomes

Cited by in corpus (12)

• A Guide to Feature Importance Methods for Scientific Inference
• Efficient Estimation Under Data Fusion
• Rank-transformed subsampling: inference for multiple data splitting and exchangeable p-values
• Assessing variable importance in survival analysis using machine learning
• A nonparametric framework for treatment effect modifier discovery in high dimensions
• Variable importance measures for heterogeneous treatment effects
• Factor Importance Ranking and Selection using Total Indices
• Algorithm-agnostic significance testing in supervised learning with multimodal data
• Flexible variable selection in the presence of missing data
• MMD-based Variable Importance for Distributional Random Forest
• Inference on summaries of a model-agnostic longitudinal variable importance trajectory with application to suicide prevention
• Hierarchical Variable Importance with Statistical Control for Medical Data-Based Prediction