A Unified Framework for Inference with General Missingness Patterns and Machine Learning Imputation

Abstract

Predictions from machine learning (ML) models are increasingly used to impute missing data, but their naive use risks biased inference. Existing methods provide valid inference with ML imputations regardless of prediction quality to enhance efficiency, while they are limited to either simple missing data structures or a missing-completely-at-random assumption. We develop a novel framework for valid statistical inference in Z-estimation problems using ML imputations under a missing-at-random assumption and for general missingness patterns. The method stratifies data by distinct missingness patterns and constructs an estimator by aggregating pattern-specific observed and prediction information. Theoretically, we establish the asymptotic properties and globally optimal aggregation that guarantees efficiency dominance over weighted complete-case analyses (WCCA). Practically, the method affords simple implementations by leveraging existing WCCA software. Extensive simulations validate theoretical results and show clear competitiveness in various scenarios. An analysis of All of Us data further demonstrates the utility of the method. The paper concludes with a brief discussion on practical implications and potential future directions.