Restricted Boltzmann Machine, recent advances and mean-field theory

Abstract

This review deals with Restricted Boltzmann Machine (RBM) under the light of statistical physics. The RBM is a classical family of Machine learning (ML) models which played a central role in the development of deep learning. Viewing it as a Spin Glass model and exhibiting various links with other models of statistical physics, we gather recent results dealing with mean-field theory in this context. First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM leading in particular to identify a {\it compositional phase} where a small number of features or modes are combined to form complex patterns. Then we discuss recent works either able to devise mean-field based learning algorithms; either able to reproduce generic aspects of the learning process from some {\it ensemble dynamics equations} or/and from linear stability arguments.

44 pages, 13 figures. Accepted for CPB

References in corpus (6)

• Mean-field message-passing equations in the Hopfield model and its generalizations
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Legendre Equivalences of Spherical Boltzmann Machines
• Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines

Cited by in corpus (11)

• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• p-Adic Statistical Field Theory and Deep Belief Networks
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Free Energy Evaluation Using Marginalized Annealed Importance Sampling
• Explaining the effects of non-convergent sampling in the training of Energy-Based Models
• Relevant Analytic Spontaneous Magnetization Relation for the Face-Centered-Cubic Ising Lattice
• Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
• Composite Spatial Monte Carlo Integration Based on Generalized Least Squares
• Optimizing Connectivity through Network Gradients for Restricted Boltzmann Machines
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data