Statistical Mechanics of Dictionary Learning

Abstract

Finding a basis matrix (dictionary) by which objective signals are represented sparsely is of major relevance in various scientific and technological fields. We consider a problem to learn a dictionary from a set of training signals. We employ techniques of statistical mechanics of disordered systems to evaluate the size of the training set necessary to typically succeed in the dictionary learning. The results indicate that the necessary size is much smaller than previously estimated, which theoretically supports and/or encourages the use of dictionary learning in practical situations.

6 pages, 4 figures

References in corpus (3)

• Statistical physics-based reconstruction in compressed sensing
• Hiding Quiet Solutions in Random Constraint Satisfaction Problems
• Replica symmetry breaking, complexity and spin representation in the generalized random energy model

Cited by in corpus (16)

• Machine learning and the physical sciences
• Statistical physics of inference: Thresholds and algorithms
• Cross validation in LASSO and its acceleration
• Phase Diagram and Approximate Message Passing for Blind Calibration and Dictionary Learning
• Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising
• Statistical limits of dictionary learning: random matrix theory and the spectral replica method
• Sparse approximation based on a random overcomplete basis
• Sparse approximation problem: how rapid simulated annealing succeeds and fails
• Evaluation of Generalized Degrees of Freedom for Sparse Estimation by Replica Method
• Sample Complexity of Bayesian Optimal Dictionary Learning
• Approximate Method of Variational Bayesian Matrix Factorization/Completion with Sparse Prior
• Some observations on the ambivalent role of symmetries in Bayesian inference problems
• Automatic Hyperparameter Tuning in Sparse Matrix Factorization
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation