High-temperature Expansions and Message Passing Algorithms

Abstract

Improved mean-field technics are a central theme of statistical physics methods applied to inference and learning. We revisit here some of these methods using high-temperature expansions for disordered systems initiated by Plefka, Georges and Yedidia. We derive the Gibbs free entropy and the subsequent self-consistent equations for a generic class of statistical models with correlated matrices and show in particular that many classical approximation schemes, such as adaptive TAP, Expectation-Consistency, or the approximations behind the Vector Approximate Message Passing algorithm all rely on the same assumptions, that are also at the heart of high-temperature expansions. We focus on the case of rotationally invariant random coupling matrices in the `high-dimensional' limit in which the number of samples and the dimension are both large, but with a fixed ratio. This encapsulates many widely studied models, such as Restricted Boltzmann Machines or Generalized Linear Models with correlated data matrices. In this general setting, we show that all the approximation schemes described before are equivalent, and we conjecture that they are exact in the thermodynamic limit in the replica symmetric phases. We achieve this conclusion by resummation of the infinite perturbation series, which generalizes a seminal result of Parisi and Potters. A rigorous derivation of this conjecture is an interesting mathematical challenge. On the way to these conclusions, we uncover several diagrammatical results in connection with free probability and random matrix theory, that are interesting independently of the rest of our work.

59 pages, updated version matching the version published in J. Stat. Mech. Correction of typos in the last version

References in corpus (25)

• Message Passing Algorithms for Compressed Sensing
• Integration with respect to the Haar measure on unitary, orthogonal and symplectic group
• Statistical physics of inference: Thresholds and algorithms
• Replica Field Theory for Deterministic Models (II): A Non-Random Spin Glass with Glassy Behavior
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Replica Field Theory for Deterministic Models: Binary Sequences with Low Autocorrelation
• Mean-Field Equations for Spin Models with Orthogonal Interaction Matrices
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Tractable approximations for probabilistic models: The adaptive TAP mean field approach
• A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
• Equilibrium Fluctuations in Maximally Noisy Extended Quantum Systems
• Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels
• Variational Free Energies for Compressed Sensing
• Mean-field theory for a spin-glass model of neural networks: TAP free energy and paramagnetic to spin-glass transition
• The jamming transition in high dimension: an analytical study of the TAP equations and the effective thermodynamic potential
• The role of the interaction matrix in mean-field spin glasses
• Perceptron capacity revisited: classification ability for correlated patterns
• Memory-free dynamics for the TAP equations of Ising models with arbitrary rotation invariant ensembles of random coupling matrices
• New scaling of Itzykson-Zuber integrals
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• Adaptive Thouless-Anderson-Palmer approach to inverse Ising problems with quenched random fields
• Higher order corrections to the effective potential close to the jamming transition in the perceptron model
• Resummed mean-field inference for strongly coupled data
• Self-Averaging Expectation Propagation

Cited by in corpus (22)

• Eigenstate Thermalization Hypothesis and Free Probability
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Mean-field inference methods for neural networks
• Full Eigenstate Thermalization via Free Cumulants in Quantum Lattice Systems
• Perturbative construction of mean-field equations in extensive-rank matrix factorization and denoising
• Statistical limits of dictionary learning: random matrix theory and the spectral replica method
• Unsupervised hierarchical clustering using the learning dynamics of RBMs
• Solving the fully-connected spherical $p$-spin model with the cavity method: equivalence with the replica results
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• Matrix denoising: Bayes-optimal estimators via low-degree polynomials
• Annealed averages in spin and matrix models
• Large deviations of extreme eigenvalues of generalized sample covariance matrices
• Random matrix universality in dynamical correlation functions at late times
• Small-Coupling Dynamic Cavity: a Bayesian mean-field framework for epidemic inference
• Out-of-equilibrium Eigenstate Thermalization Hypothesis
• Structured random matrices and cyclic cumulants: A free probability approach
• Probes of Full Eigenstate Thermalization in Ergodicity-Breaking Quantum Circuits
• Eigenstate Thermalization Hypothesis and Random Matrix Theory Universality in Few-Body Systems
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation
• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Diagrammatics of free energies with fixed variance for high-dimensional data
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation