Low Rank Approximation with Entrywise ${\ell }_1$-Norm Error

Abstract

We study the ${\ell }_1$-low rank approximation problem, where for a given $n\times d$ matrix $A$ and approximation factor $\alpha \ge 1$, the goal is to output a rank-$k$ matrix $\hat{A}$ for which $$\Vert A-\hat{A}{\Vert }_1\le \alpha \cdot \underset{\text{rank-}k\text{ matrices}{A}^{\prime }}{\min }\Vert A-A^{\prime }{\Vert }_1,$$ where for an $n\times d$ matrix $C$, we let $\Vert C{\Vert }_1={\sum }_{i=1}^n{\sum }_{j=1}^d|C_{i,j}|$. This error measure is known to be more robust than the Frobenius norm in the presence of outliers and is indicated in models where Gaussian assumptions on the noise may not apply. The problem was shown to be NP-hard by Gillis and Vavasis and a number of heuristics have been proposed. It was asked in multiple places if there are any approximation algorithms. We give the first provable approximation algorithms for ${\ell }_1$-low rank approximation, showing that it is possible to achieve approximation factor $\alpha =(\log d)\cdot \mathrm{poly}(k)$ in $\mathrm{nnz}(A)+(n+d)\mathrm{poly}(k)$ time, where $\mathrm{nnz}(A)$ denotes the number of non-zero entries of $A$. If $k$ is constant, we further improve the approximation ratio to $O(1)$ with a $\mathrm{poly}(nd)$-time algorithm. Under the Exponential Time Hypothesis, we show there is no $\mathrm{poly}(nd)$-time algorithm achieving a $(1+\frac{1}{{\log }^{1+\gamma }(nd)})$-approximation, for $\gamma >0$ an arbitrarily small constant, even when $k=1$. We give a number of additional results for ${\ell }_1$-low rank approximation: nearly tight upper and lower bounds for column subset selection, CUR decompositions, extensions to low rank approximation with respect to ${\ell }_p$-norms for $1\le p<2$ and earthmover distance, low-communication distributed protocols and low-memory streaming algorithms, algorithms with limited randomness, and bicriteria algorithms. We also give a preliminary empirical evaluation.

STOC 2017

References in corpus (4)

• Non-convex Robust PCA
• Improved Distributed Principal Component Analysis
• Algorithms in Real Algebraic Geometry: A Survey
• Iteratively Reweighted Least Squares Algorithms for L1-Norm Principal Component Analysis

Cited by in corpus (9)

• Towards Fast Computation of Certified Robustness for ReLU Networks
• Recovery Guarantees for One-hidden-layer Neural Networks
• Introduction to Nonnegative Matrix Factorization
• On the Gap Between Strict-Saddles and True Convexity: An Omega(log d) Lower Bound for Eigenvector Approximation
• An Improved Cutting Plane Method for Convex Optimization, Convex-Concave Games and its Applications
• Sketching for Kronecker Product Regression and P-splines
• Nonlinear Inductive Matrix Completion based on One-layer Neural Networks
• Streaming Balanced Clustering
• Sketching Transformed Matrices with Applications to Natural Language Processing