Spectral Dynamics of Learning Restricted Boltzmann Machines

Abstract

The Restricted Boltzmann Machine (RBM), an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Starting from empirical observations, we propose a generic statistical ensemble for the weight matrix of the RBM and characterize its mean evolution. This let us show how in the linear regime, in which the RBM is found to operate at the beginning of the training, the statistical properties of the data drive the selection of the unstable modes of the weight matrix. A set of equations characterizing the non-linear regime is then derived, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the RBM.

References in corpus (9)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Statistical physics of inference: Thresholds and algorithms
• Multitasking associative networks
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Mean-Field Equations for Spin Models with Orthogonal Interaction Matrices
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Mean-Field Inference in Gaussian Restricted Boltzmann Machine

Cited by in corpus (30)

• Machine learning and the physical sciences
• Machine Learning for Condensed Matter Physics
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Machine Learning Spatial Geometry from Entanglement Features
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Mean-field inference methods for neural networks
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Unsupervised hierarchical clustering using the learning dynamics of RBMs
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Minimal model of permutation symmetry in unsupervised learning
• Inferring effective couplings with Restricted Boltzmann Machines
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
• Efficient training of energy-based models via spin-glass control
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Gaussian-Spherical Restricted Boltzmann Machines
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• A Probability Density Theory for Spin-Glass Systems
• A relativistic extension of Hopfield neural networks via the mechanical analogy
• Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
• Explaining the effects of non-convergent sampling in the training of Energy-Based Models
• On the role of non-linear latent features in bipartite generative neural networks
• Nonequilibrium thermodynamics of self-supervised learning
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data
• Saddle Hierarchy in Dense Associative Memory