A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines

Abstract

We define a message-passing algorithm for computing magnetizations in Restricted Boltzmann machines, which are Ising models on bipartite graphs introduced as neural network models for probability distributions over spin configurations. To model nontrivial statistical dependencies between the spins' couplings, we assume that the rectangular coupling matrix is drawn from an arbitrary bi-rotation invariant random matrix ensemble. Using the dynamical functional method of statistical mechanics we exactly analyze the dynamics of the algorithm in the large system limit. We prove the global convergence of the algorithm under a stability criterion and compute asymptotic convergence rates showing excellent agreement with numerical simulations.

arXiv comment29 pages, 2 figures

References in corpus (14)

• The dynamics of message passing on dense graphs, with applications to compressed sensing
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• A Theory of Solving TAP Equations for Ising Models with General Invariant Random Matrices
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Inference from correlated patterns: a unified theory for perceptron learning and linear vector channels
• High-temperature Expansions and Message Passing Algorithms
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Mean-field inference methods for neural networks
• New scaling of Itzykson-Zuber integrals
• Memory-free dynamics for the TAP equations of Ising models with arbitrary rotation invariant ensembles of random coupling matrices
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• Integration of invariant matrices and application to statistics
• Analysis of Bayesian Inference Algorithms by the Dynamical Functional Approach

Cited by in corpus (3)

• Dynamics of coordinate ascent variational inference: A case study in 2D Ising models
• Macroscopic Analysis of Vector Approximate Message Passing in a Model Mismatch Setting
• Exact solution to the random sequential dynamics of a message passing algorithm