Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors

Abstract

Restricted Boltzmann Machines are described by the Gibbs measure of a bipartite spin glass, which in turn corresponds to the one of a generalised Hopfield network. This equivalence allows us to characterise the state of these systems in terms of retrieval capabilities, both at low and high load. We study the paramagnetic-spin glass and the spin glass-retrieval phase transitions, as the pattern (i.e. weight) distribution and spin (i.e. unit) priors vary smoothly from Gaussian real variables to Boolean discrete variables. Our analysis shows that the presence of a retrieval phase is robust and not peculiar to the standard Hopfield model with Boolean patterns. The retrieval region is larger when the pattern entries and retrieval units get more peaked and, conversely, when the hidden units acquire a broader prior and therefore have a stronger response to high fields. Moreover, at low load retrieval always exists below some critical temperature, for every pattern distribution ranging from the Boolean to the Gaussian case.

18 pages, 9 figures; typos added

References in corpus (7)

• Mean-field message-passing equations in the Hopfield model and its generalizations
• Extensive load in multitasking associative networks
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Free energy and complexity of spherical bipartite models
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Legendre Duality of Spherical and Gaussian Spin Glasses

Cited by in corpus (42)

• Machine learning and the physical sciences
• Deep Learning in Spiking Neural Networks
• Path Integral Approach to Random Neural Networks
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Quantum field-theoretic machine learning
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Mean-field inference methods for neural networks
• Neural networks with redundant representation: detecting the undetectable
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Non-Convex Multi-species Hopfield models
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Dreaming neural networks: rigorous results
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Learning a Restricted Boltzmann Machine using biased Monte Carlo sampling
• Minimal model of permutation symmetry in unsupervised learning
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Inferring effective couplings with Restricted Boltzmann Machines
• Legendre Equivalences of Spherical Boltzmann Machines
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• Long anisotropic absolute refractory periods with rapid rise-times to reliable responsiveness
• Space of Functions Computed by Deep-Layered Machines
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Gaussian-Spherical Restricted Boltzmann Machines
• Dense Hopfield Networks in the Teacher-Student Setting
• Riemann-Theta Boltzmann Machine
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
• Variational mean-field theory for training restricted Boltzmann machines with binary synapses
• Dataset-Free Weight-Initialization on Restricted Boltzmann Machine
• Equivalence between algorithmic instability and transition to replica symmetry breaking in perceptron learning systems
• A generalized Hopfield model to store and retrieve mismatched memory patterns
• Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
• On the role of non-linear latent features in bipartite generative neural networks
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data
• Saddle Hierarchy in Dense Associative Memory