Naive imputation implicitly regularizes high-dimensional linear models

Abstract

Two different approaches exist to handle missing values for prediction: either imputation, prior to fitting any predictive algorithms, or dedicated methods able to natively incorporate missing values. While imputation is widely (and easily) use, it is unfortunately biased when low-capacity predictors (such as linear models) are applied afterward. However, in practice, naive imputation exhibits good predictive performance. In this paper, we study the impact of imputation in a high-dimensional linear model with MCAR missing data. We prove that zero imputation performs an implicit regularization closely related to the ridge method, often used in high-dimensional problems. Leveraging on this connection, we establish that the imputation bias is controlled by a ridge bias, which vanishes in high dimension. As a predictor, we argue in favor of the averaged SGD strategy, applied to zero-imputed data. We establish an upper bound on its generalization error, highlighting that imputation is benign in the d $\sqrt$ n regime. Experiments illustrate our findings.

References in corpus (12)

• Scikit-learn: Machine Learning in Python
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• GAIN: Missing Data Imputation using Generative Adversarial Nets
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Random design analysis of ridge regression
• Missing Data Imputation using Optimal Transport
• Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss
• Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices
• Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression
• NeuMiss networks: differentiable programming for supervised learning with missing values
• On Robustness of Principal Component Regression
• Linear predictor on linearly-generated data with missing values: non consistency and solutions

Cited by in corpus (1)

• Human spatial dynamics for electricity demand forecasting: the case of France during the 2022 energy crisis