Thermodynamics of Restricted Boltzmann Machines and related learning dynamics

Abstract

We investigate the thermodynamic properties of a Restricted Boltzmann Machine (RBM), a simple energy-based generative model used in the context of unsupervised learning. Assuming the information content of this model to be mainly reflected by the spectral properties of its weight matrix $W$, we try to make a realistic analysis by averaging over an appropriate statistical ensemble of RBMs. First, a phase diagram is derived. Otherwise similar to that of the Sherrington- Kirkpatrick (SK) model with ferromagnetic couplings, the RBM's phase diagram presents a ferromagnetic phase which may or may not be of compositional type depending on the kurtosis of the distribution of the components of the singular vectors of $W$. Subsequently, the learning dynamics of the RBM is studied in the thermodynamic limit. A "typical" learning trajectory is shown to solve an effective dynamical equation, based on the aforementioned ensemble average and explicitly involving order parameters obtained from the thermodynamic analysis. In particular, this let us show how the evolution of the dominant singular values of $W$, and thus of the unstable modes, is driven by the input data. At the beginning of the training, in which the RBM is found to operate in the linear regime, the unstable modes reflect the dominant covariance modes of the data. In the non-linear regime, instead, the selected modes interact and eventually impose a matching of the order parameters to their empirical counterparts estimated from the data. Finally, we illustrate our considerations by performing experiments on both artificial and real data, showing in particular how the RBM operates in the ferromagnetic compositional phase.

35 pages, 9 figures

References in corpus (3)

• Mean-field message-passing equations in the Hopfield model and its generalizations
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Advanced Mean Field Theory of Restricted Boltzmann Machine

Cited by in corpus (32)

• Machine Learning for Condensed Matter Physics
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
• Mean-field inference methods for neural networks
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Unsupervised hierarchical clustering using the learning dynamics of RBMs
• Quantifying Relevance in Learning and Inference
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Inferring effective couplings with Restricted Boltzmann Machines
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• Gaussian-Spherical Restricted Boltzmann Machines
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• The effect of priors on Learning with Restricted Boltzmann Machines
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• Interpretable machine learning of amino acid patterns in proteins: a statistical ensemble approach
• Thermodynamics of the Ising model encoded in restricted Boltzmann machines
• Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines
• A Probability Density Theory for Spin-Glass Systems
• A Thermodynamic Core using Voltage-Controlled Spin-Orbit-Torque Magnetic Tunnel Junctions
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
• On the role of non-linear latent features in bipartite generative neural networks
• Explaining the effects of non-convergent sampling in the training of Energy-Based Models
• Self-regularizing restricted Boltzmann machines
• Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data
• Neuromodulation via Krotov-Hopfield Improves Accuracy and Robustness of RBMs
• Maximal Relevance and Optimal Learning Machines
• Saddle Hierarchy in Dense Associative Memory