Floodgate: inference for model-free variable importance

Abstract

Many modern applications seek to understand the relationship between an outcome variable $Y$ and a covariate $X$ in the presence of a (possibly high-dimensional) confounding variable $Z$. Although much attention has been paid to testing \emph{whether} $Y$ depends on $X$ given $Z$, in this paper we seek to go beyond testing by inferring the \emph{strength} of that dependence. We first define our estimand, the minimum mean squared error (mMSE) gap, which quantifies the conditional relationship between $Y$ and $X$ in a way that is deterministic, model-free, interpretable, and sensitive to nonlinearities and interactions. We then propose a new inferential approach called \emph{floodgate} that can leverage any working regression function chosen by the user (allowing, e.g., it to be fitted by a state-of-the-art machine learning algorithm or be derived from qualitative domain knowledge) to construct asymptotic confidence bounds, and we apply it to the mMSE gap. \acc{We additionally show that floodgate's accuracy (distance from confidence bound to estimand) is adaptive to the error of the working regression function.} We then show we can apply the same floodgate principle to a different measure of variable importance when $Y$ is binary. Finally, we demonstrate floodgate's performance in a series of simulations and apply it to data from the UK Biobank to infer the strengths of dependence of platelet count on various groups of genetic mutations.

References in corpus (13)

• Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
• Valid post-selection inference
• Understanding Global Feature Contributions With Additive Importance Measures
• Statistical significance in high-dimensional linear models
• Gene Hunting with Knockoffs for Hidden Markov Models
• Confidence sets in sparse regression
• Causal Inference in Genetic Trio Studies
• High-dimensional inference in misspecified linear models
• Exact Post-Selection Inference for Sequential Regression Procedures
• Bootstrapping and Sample Splitting For High-Dimensional, Assumption-Free Inference
• Discussion: "A significance test for the lasso"
• Fast and Powerful Conditional Randomization Testing via Distillation
• On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning

Cited by in corpus (4)

• Cross-validation: what does it estimate and how well does it do it?
• Relating the Partial Dependence Plot and Permutation Feature Importance to the Data Generating Process
• Kernel Partial Correlation Coefficient -- a Measure of Conditional Dependence
• Decorrelated Variable Importance