In-Database Data Imputation

Abstract

Missing data is a widespread problem in many domains, creating challenges in data analysis and decision making. Traditional techniques for dealing with missing data, such as excluding incomplete records or imputing simple estimates (e.g., mean), are computationally efficient but may introduce bias and disrupt variable relationships, leading to inaccurate analyses. Model-based imputation techniques offer a more robust solution that preserves the variability and relationships in the data, but they demand significantly more computation time, limiting their applicability to small datasets. This work enables efficient, high-quality, and scalable data imputation within a database system using the widely used MICE method. We adapt this method to exploit computation sharing and a ring abstraction for faster model training. To impute both continuous and categorical values, we develop techniques for in-database learning of stochastic linear regression and Gaussian discriminant analysis models. Our MICE implementations in PostgreSQL and DuckDB outperform alternative MICE implementations and model-based imputation techniques by up to two orders of magnitude in terms of computation time, while maintaining high imputation quality.

arXiv commentPublished at SIGMOD 2024 (26 pages)

References in corpus (5)

• Scikit-learn: Machine Learning in Python
• GAIN: Missing Data Imputation using Generative Adversarial Nets
• Handling Missing Data with Graph Representation Learning
• MIRACLE: Causally-Aware Imputation via Learning Missing Data Mechanisms
• Missing Data Imputation and Acquisition with Deep Hierarchical Models and Hamiltonian Monte Carlo