Restricted Boltzmann Machine, recent advances and mean-field theory

Abstract

This review deals with Restricted Boltzmann Machine (RBM) under the light of statistical physics. The RBM is a classical family of Machine learning (ML) models which played a central role in the development of deep learning. Viewing it as a Spin Glass model and exhibiting various links with other models of statistical physics, we gather recent results dealing with mean-field theory in this context. First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM leading in particular to identify a {\it compositional phase} where a small number of features or modes are combined to form complex patterns. Then we discuss recent works either able to devise mean-field based learning algorithms; either able to reproduce generic aspects of the learning process from some {\it ensemble dynamics equations} or/and from linear stability arguments.

arXiv comment44 pages, 13 figures. Accepted for CPB

References in corpus (32)

• Exchange Monte Carlo Method and Application to Spin Glass Simulations
• Machine learning phases of matter
• A high-bias, low-variance introduction to Machine Learning for physicists
• Multitasking associative networks
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Mean-field message-passing equations in the Hopfield model and its generalizations
• The Bethe approximation for solving the inverse Ising problem: a comparison with other inference methods
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Finite Connectivity Attractor Neural Networks
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Selection of sequence motifs and generative Hopfield-Potts models for protein familiesilies
• High-temperature Expansions and Message Passing Algorithms
• Bethe-Peierls approximation and the inverse Ising model
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Replica Cluster Variational Method: the Replica Symmetric solution for the 2D random bond Ising model
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• Legendre Equivalences of Spherical Boltzmann Machines
• Mean-Field Inference in Gaussian Restricted Boltzmann Machine
• Training Restricted Boltzmann Machines via the Thouless-Anderson-Palmer Free Energy
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• Restricted Boltzmann Machine with Multivalued Hidden Variables: a model suppressing over-fitting
• Gaussian-Spherical Restricted Boltzmann Machines
• Role of zero synapses in unsupervised feature learning
• Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Inverse problems for structured datasets using parallel TAP equations and RBM

Cited by in corpus (11)

• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• p-Adic Statistical Field Theory and Deep Belief Networks
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Free Energy Evaluation Using Marginalized Annealed Importance Sampling
• Explaining the effects of non-convergent sampling in the training of Energy-Based Models
• Relevant Analytic Spontaneous Magnetization Relation for the Face-Centered-Cubic Ising Lattice
• Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
• Composite Spatial Monte Carlo Integration Based on Generalized Least Squares
• Optimizing Connectivity through Network Gradients for Restricted Boltzmann Machines
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data