New Algorithms for Learning Incoherent and Overcomplete Dictionaries

Abstract

In sparse recovery we are given a matrix $A$ (the dictionary) and a vector of the form $AX$ where $X$ is sparse, and the goal is to recover $X$. This is a central notion in signal processing, statistics and machine learning. But in applications such as sparse coding, edge detection, compression and super resolution, the dictionary $A$ is unknown and has to be learned from random examples of the form $Y=AX$ where $X$ is drawn from an appropriate distribution --- this is the dictionary learning problem. In most settings, $A$ is overcomplete: it has more columns than rows. This paper presents a polynomial-time algorithm for learning overcomplete dictionaries; the only previously known algorithm with provable guarantees is the recent work of Spielman, Wang and Wright who gave an algorithm for the full-rank case, which is rarely the case in applications. Our algorithm applies to incoherent dictionaries which have been a central object of study since they were introduced in seminal work of Donoho and Huo. In particular, a dictionary is $\mu$-incoherent if each pair of columns has inner product at most $\mu /\sqrt{n}$. The algorithm makes natural stochastic assumptions about the unknown sparse vector $X$, which can contain $k\le c\min (\sqrt{n}/\mu \log n,m^{1/2-\eta })$ non-zero entries (for any $\eta >0$). This is close to the best $k$ allowable by the best sparse recovery algorithms even if one knows the dictionary $A$ exactly. Moreover, both the running time and sample complexity depend on $\log 1/\epsilon$, where $\epsilon$ is the target accuracy, and so our algorithms converge very quickly to the true dictionary. Our algorithm can also tolerate substantial amounts of noise provided it is incoherent with respect to the dictionary (e.g., Gaussian). In the noisy setting, our running time and sample complexity depend polynomially on $1/\epsilon$, and this is necessary.

References in corpus (5)

• Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition
• A Spectral Algorithm for Latent Dirichlet Allocation
• Polynomial Learning of Distribution Families
• Learning Topic Models - Going beyond SVD
• A Clustering Approach to Learn Sparsely-Used Overcomplete Dictionaries

Cited by in corpus (21)

• Non-convex Optimization for Machine Learning
• Provable Bounds for Learning Some Deep Representations
• Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture
• Phase transitions and sample complexity in Bayes-optimal matrix factorization
• When Are Nonconvex Problems Not Scary?
• Efficient Sum of Outer Products Dictionary Learning (SOUP-DIL) and Its Application to Inverse Problems
• Minimax Lower Bounds on Dictionary Learning for Tensor Data
• Sparse Matrix Factorization
• Local Identification of Overcomplete Dictionaries
• Learning Mixtures of Separable Dictionaries for Tensor Data: Analysis and Algorithms
• Convolutional Phase Retrieval via Gradient Descent
• Dictionary Learning from Incomplete Data
• Identifiability of Kronecker-structured Dictionaries for Tensor Data
• Dictionary Learning with BLOTLESS Update
• Performance Limits of Dictionary Learning for Sparse Coding
• Efficient Sum of Outer Products Dictionary Learning (SOUP-DIL) - The ${\ell }_0$ Method
• Tensor Graphical Model: Non-convex Optimization and Statistical Inference
• Alternating minimization for dictionary learning: Local Convergence Guarantees
• A Compressed Sampling and Dictionary Learning Framework for WDM-Based Distributed Fiber Sensing
• Combinatorial rigidity of Incidence systems and Application to Dictionary learning
• On the Minimax Risk of Dictionary Learning