Inference in High Dimensions with the Penalized Score Test

Abstract

In recent years, there has been considerable theoretical development regarding variable selection consistency of penalized regression techniques, such as the lasso. However, there has been relatively little work on quantifying the uncertainty in these selection procedures. In this paper, we propose a new method for inference in high dimensions using a score test based on penalized regression. In this test, we perform penalized regression of an outcome on all but a single feature, and test for correlation of the residuals with the held-out feature. This procedure is applied to each feature in turn. Interestingly, when an ${\ell }_1$ penalty is used, the sparsity pattern of the lasso corresponds exactly to a decision based on the proposed test. Further, when an ${\ell }_2$ penalty is used, the test corresponds precisely to a score test in a mixed effects model, in which the effects of all but one feature are assumed to be random. We formulate the hypothesis being tested as a compromise between the null hypotheses tested in simple linear regression on each feature and in multiple linear regression on all features, and develop reference distributions for some well-known penalties. We also examine the behavior of the test on real and simulated data.

32 pages, 5 figures

References in corpus (2)

• Exact Post-Selection Inference for Sequential Regression Procedures
• On the Distribution of Penalized Maximum Likelihood Estimators: The LASSO, SCAD, and Thresholding

Cited by in corpus (10)

• Controlling the false discovery rate via knockoffs
• Valid Post-Selection and Post-Regularization Inference: An Elementary, General Approach
• Goodness of fit tests for high-dimensional linear models
• A General Theory of Hypothesis Tests and Confidence Regions for Sparse High Dimensional Models
• A Likelihood Ratio Framework for High Dimensional Semiparametric Regression
• Derandomizing Knockoffs
• Using Regression Kernels to Forecast A Failure to Appear in Court
• Uncertainty Quantification Under Group Sparsity
• Estimator Augmentation with Applications in High-Dimensional Group Inference
• Testing for high-dimensional network parameters in auto-regressive models