Trust, but verify: benefits and pitfalls of least-squares refitting in high dimensions

Abstract

Least-squares refitting is widely used in high dimensional regression to reduce the prediction bias of l1-penalized estimators (e.g., Lasso and Square-Root Lasso). We present theoretical and numerical results that provide new insights into the benefits and pitfalls of least-squares refitting. In particular, we consider both prediction and estimation, and we pay close attention to the effects of correlations in the design matrices of linear regression models, since these correlations - although often neglected - are crucial in the context of linear regression, especially in high dimensional contexts. First, we demonstrate that the benefit of least-squares refitting strongly depends on the setting and task under consideration: least-squares refitting can be beneficial even for settings with highly correlated design matrices but is not advisable for all settings, and least-squares refitting can be beneficial for estimation but performs better for prediction. Finally, we introduce a criterion that indicates whether least-squares refitting is advisable for a specific setting and task under consideration, and we conduct a thorough simulation study involving the Lasso to show the usefulness of this criterion.

Cited by in corpus (7)

• Efficient Smoothed Concomitant Lasso Estimation for High Dimensional Regression
• On estimation of the diagonal elements of a sparse precision matrix
• Tractable Post-Selection Maximum Likelihood Inference for the Lasso
• Bias-Reduction in Variational Regularization
• Block based refitting in ${\ell }_{12}$ sparse regularisation
• Refitting solutions promoted by ${\ell }_{12}$ sparse analysis regularization with block penalties
• On debiasing restoration algorithms: applications to total-variation and nonlocal-means