Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising

Abstract

Factorization of matrices where the rank of the two factors diverges linearly with their sizes has many applications in diverse areas such as unsupervised representation learning, dictionary learning or sparse coding. We consider a setting where the two factors are generated from known component-wise independent prior distributions, and the statistician observes a (possibly noisy) component-wise function of their matrix product. In the limit where the dimensions of the matrices tend to infinity, but their ratios remain fixed, we expect to be able to derive closed form expressions for the optimal mean squared error on the estimation of the two factors. However, this remains a very involved mathematical and algorithmic problem. A related, but simpler, problem is extensive-rank matrix denoising, where one aims to reconstruct a matrix with extensive but usually small rank from noisy measurements. In this paper, we approach both these problems using high-temperature expansions at fixed order parameters. This allows to clarify how previous attempts at solving these problems failed at finding an asymptotically exact solution. We provide a systematic way to derive the corrections to these existing approximations, taking into account the structure of correlations particular to the problem. Finally, we illustrate our approach in detail on the case of extensive-rank matrix denoising. We compare our results with known optimal rotationally-invariant estimators, and show how exact asymptotic calculations of the minimal error can be performed using extensive-rank matrix integrals.

30 pages (main text), 25 pages of references and appendices. v2: Adding clarifications and a new result to derive the optimal denoising estimator from the asymptotic free energy. v3: corrections to match the published version

References in corpus (7)

• Expectation Propagation for approximate Bayesian inference
• Cleaning large correlation matrices: tools from random matrix theory
• Large deviations and stochastic calculus for large random matrices
• Statistical limits of dictionary learning: random matrix theory and the spectral replica method
• Deep learning via message passing algorithms based on belief propagation
• Large Deviations Asymptotics of Rectangular Spherical Integral
• Graph-based Approximate Message Passing Iterations

Cited by in corpus (15)

• Statistical limits of dictionary learning: random matrix theory and the spectral replica method
• Depth induces scale-averaging in overparameterized linear Bayesian neural networks
• Matrix factorization with neural networks
• Deep learning via message passing algorithms based on belief propagation
• Matrix denoising: Bayes-optimal estimators via low-degree polynomials
• Singular Vectors of Sums of Rectangular Random Matrices and Optimal Estimators of High-Rank Signals: The Extensive Spike Model
• Bayesian reconstruction of memories stored in neural networks from their connectivity
• Spherical Integrals of Sublinear Rank
• Some observations on the ambivalent role of symmetries in Bayesian inference problems
• Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokens
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• A Two-HCIZ Gaussian Matrix Model for Non-intersecting Brownian Bridges
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Diagrammatics of free energies with fixed variance for high-dimensional data