Phase transitions in Restricted Boltzmann Machines with generic priors

Abstract

We study Generalised Restricted Boltzmann Machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as Generalised Hopfield models. We underline the role of the retrieval phase for both inference and learning processes and we show that retrieval is robust for a large class of weight and unit priors, beyond the standard Hopfield scenario. Furthermore we show how the paramagnetic phase boundary is directly related to the optimal size of the training set necessary for good generalisation in a teacher-student scenario of unsupervised learning.

5 pages, 4 figures; extensive simulations and 2 new figures added; corrected typos; added references

References in corpus (4)

• Mean-field message-passing equations in the Hopfield model and its generalizations
• Extensive load in multitasking associative networks
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Legendre Duality of Spherical and Gaussian Spin Glasses

Cited by in corpus (39)

• Deep Learning in Spiking Neural Networks
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Quantum field-theoretic machine learning
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Mean-field inference methods for neural networks
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Non-Convex Multi-species Hopfield models
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Dreaming neural networks: rigorous results
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• Minimal model of permutation symmetry in unsupervised learning
• Entanglement Features of Random Neural Network Quantum States
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• Legendre Equivalences of Spherical Boltzmann Machines
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• Unified CNNs and transformers underlying learning mechanism reveals multi-head attention modus vivendi
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Dense Hopfield Networks in the Teacher-Student Setting
• Gaussian-Spherical Restricted Boltzmann Machines
• Riemann-Theta Boltzmann Machine
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Overlap synchronisation in multipartite random energy models
• A generalized Hopfield model to store and retrieve mismatched memory patterns
• Criticality & Deep Learning II: Momentum Renormalisation Group
• Criticality & Deep Learning I: Generally Weighted Nets
• Role of Delay in Brain Dynamics
• Pattern recognition in Deep Boltzmann machines
• A relativistic extension of Hopfield neural networks via the mechanical analogy
• Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks
• On the role of non-linear latent features in bipartite generative neural networks
• Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data
• Saddle Hierarchy in Dense Associative Memory
• Supervised and Unsupervised protocols for hetero-associative neural networks
• Emergence of Nonequilibrium Latent Cycles in Unsupervised Generative Modeling