Matrix Completion with Quantified Uncertainty through Low Rank Gaussian Copula
arXiv:2006.10829
Abstract
Modern large scale datasets are often plagued with missing entries. For tabular data with missing values, a flurry of imputation algorithms solve for a complete matrix which minimizes some penalized reconstruction error. However, almost none of them can estimate the uncertainty of its imputations. This paper proposes a probabilistic and scalable framework for missing value imputation with quantified uncertainty. Our model, the Low Rank Gaussian Copula, augments a standard probabilistic model, Probabilistic Principal Component Analysis, with marginal transformations for each column that allow the model to better match the distribution of the data. It naturally handles Boolean, ordinal, and real-valued observations and quantifies the uncertainty in each imputation. The time required to fit the model scales linearly with the number of rows and the number of columns in the dataset. Empirical results show the method yields state-of-the-art imputation accuracy across a wide range of data types, including those with high rank. Our uncertainty measure predicts imputation error well: entries with lower uncertainty do have lower imputation error (on average). Moreover, for real-valued data, the resulting confidence intervals are well-calibrated.
arXiv commentAccepted by NeurIPS 2020
References in corpus (7)
- Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
- Inference and Uncertainty Quantification for Noisy Matrix Completion
- OBOE: Collaborative Filtering for AutoML Model Selection
- Algebraic Variety Models for High-Rank Matrix Completion
- Exponential Family Matrix Completion under Structural Constraints
- XPCA: Extending PCA for a Combination of Discrete and Continuous Variables
- Information-Guided Sampling for Low-Rank Matrix Completion