On the Local Correctness of L^1 Minimization for Dictionary Learning

Abstract

The idea that many important classes of signals can be well-represented by linear combinations of a small set of atoms selected from a given dictionary has had dramatic impact on the theory and practice of signal processing. For practical problems in which an appropriate sparsifying dictionary is not known ahead of time, a very popular and successful heuristic is to search for a dictionary that minimizes an appropriate sparsity surrogate over a given set of sample data. While this idea is appealing, the behavior of these algorithms is largely a mystery; although there is a body of empirical evidence suggesting they do learn very effective representations, there is little theory to guarantee when they will behave correctly, or when the learned dictionary can be expected to generalize. In this paper, we take a step towards such a theory. We show that under mild hypotheses, the dictionary learning problem is locally well-posed: the desired solution is indeed a local minimum of the ${\ell }^1$ norm. Namely, if $\mb A \in \Re^{m \times n}$ is an incoherent (and possibly overcomplete) dictionary, and the coefficients $\mb X \in \Re^{n \times p}$ follow a random sparse model, then with high probability $(\mb A,\mb X)$ is a local minimum of the ${\ell }^1$ norm over the manifold of factorizations $(\mb A',\mb X')$ satisfying $\mb A' \mb X' = \mb Y$, provided the number of samples $p=\Omega (n^{3}k)$. For overcomplete $\mb A$, this is the first result showing that the dictionary learning problem is locally solvable. Our analysis draws on tools developed for the problem of completing a low-rank matrix from a small subset of its entries, which allow us to overcome a number of technical obstacles; in particular, the absence of the restricted isometry property.

37 pages, 1 figure

Cited by in corpus (13)

• Simultaneous Codeword Optimization (SimCO) for Dictionary Update and Learning
• Exact Recovery of Sparsely-Used Dictionaries
• Constrained Overcomplete Analysis Operator Learning for Cosparse Signal Modelling
• New Algorithms for Learning Incoherent and Overcomplete Dictionaries
• On the Identifiability of Overcomplete Dictionaries via the Minimisation Principle Underlying K-SVD
• Noisy Matrix Completion under Sparse Factor Models
• Phase Diagram and Approximate Message Passing for Blind Calibration and Dictionary Learning
• Local stability and robustness of sparse dictionary learning in the presence of noise
• Local Identification of Overcomplete Dictionaries
• Nonparametric Basis Pursuit via Sparse Kernel-based Learning
• Dictionary Learning with BLOTLESS Update
• Sparse and spurious: dictionary learning with noise and outliers
• Conditional Sparse Coding and Grouped Multivariate Regression