Factor Importance Ranking and Selection using Total Indices

Abstract

Factor importance measures the impact of each feature on output prediction accuracy. Many existing works focus on the model-based importance, but an important feature in one learning algorithm may hold little significance in another model. Hence, a factor importance measure ought to characterize the feature's predictive potential without relying on a specific prediction algorithm. Such algorithm-agnostic importance is termed as intrinsic importance in Williamson et al. (2023), but their estimator again requires model fitting. To bypass the modeling step, we present the equivalence between predictiveness potential and total Sobol' indices from global sensitivity analysis, and introduce a novel consistent estimator that can be directly estimated from noisy data. Integrating with forward selection and backward elimination gives rise to FIRST, Factor Importance Ranking and Selection using Total (Sobol') indices. Extensive simulations are provided to demonstrate the effectiveness of FIRST on regression and binary classification problems, and a clear advantage over the state-of-the-art methods.

References in corpus (15)

• XGBoost: A Scalable Tree Boosting System
• A Unified Approach to Interpreting Model Predictions
• ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R
• Measuring and testing dependence by correlation of distances
• Interpretable machine learning: definitions, methods, and applications
• All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously
• Component selection and smoothing in multivariate nonparametric regression
• A Benchmark for Interpretability Methods in Deep Neural Networks
• Understanding Global Feature Contributions With Additive Importance Measures
• A general framework for inference on algorithm-agnostic variable importance
• Forward-Backward Selection with Early Dropping
• Efficient nonparametric statistical inference on population feature importance using Shapley values
• Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence
• Variable importance without impossible data
• Model free variable importance for high dimensional data