Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks

Abstract

Some interesting recent advances in the theoretical understanding of neural networks have been informed by results from the physics of disordered many-body systems. Motivated by these findings, this work uses the replica technique to study the mathematically tractable bipartite Sherrington-Kirkpatrick (SK) spin glass model, which is formally similar to a Restricted Boltzmann Machine (RBM) neural network. The bipartite SK model has been previously studied assuming replica symmetry; here this assumption is relaxed and a replica symmetry breaking analysis is performed. The bipartite SK model is found to have many features in common with Parisi's solution of the original, unipartite SK model, including the existence of a multitude of pure states which are related in a hierarchical, ultrametric fashion. As an application of this analysis, the optimal cost for a graph partitioning problem is shown to be simply related to the ground state energy of the bipartite SK model. As a second application, empirical investigations reveal that the Gibbs sampled outputs of an RBM trained on the MNIST data set are more ultrametrically distributed than the input data itself.

arXiv comment33 pages, 14 figures

References in corpus (26)

• Machine learning phases of matter
• The Loss Surfaces of Multilayer Networks
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• The statistics of critical points of Gaussian fields on large-dimensional spaces
• Complexity of random smooth functions on the high-dimensional sphere
• Analysis of the infinity-replica symmetry breaking solution of the Sherrington-Kirkpatrick model
• The free energy in a multi-species Sherrington-Kirkpatrick model
• Equilibrium statistical mechanics of bipartite spin systems
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• How glassy are neural networks?
• Numerical Simulations of Spin Glass Systems
• Universality in bipartite mean field spin glasses
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Lack of Ultrametricity in the Low-Temperature phase of 3D Ising Spin Glasses
• Ultrametricity and clustering of states in spin glasses: A one-dimensional view
• Mean field bipartite spin models treated with mechanical techniques
• Rigorous Results on the Bipartite Mean-Field Model
• Introduction to Monte Carlo Methods
• A Correspondence Between Random Neural Networks and Statistical Field Theory
• Non-Convex Multipartite Ferromagnets
• Riemann-Theta Boltzmann Machine
• Ultrametricity of optimal transport substates for multiple interacting paths over a square lattice network
• Glassy dynamics in deep neural networks: A structural comparison

Cited by in corpus (17)

• Restricted Boltzmann Machine, recent advances and mean-field theory
• Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions
• Free energy in multi-species mixed $p$-spin spherical models
• Replica symmetry breaking in dense neural networks
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Minimal model of permutation symmetry in unsupervised learning
• A Dynamical Mean-Field Theory for Learning in Restricted Boltzmann Machines
• Efficient training of energy-based models via spin-glass control
• Message-passing algorithm of quantum annealing with nonstoquastic Hamiltonian
• On the free energy of vector spin glasses with non-convex interactions
• Self-Supervised Learning of Generative Spin-Glasses with Normalizing Flows
• Numerical renormalization of glassy dynamics
• Dataset-Free Weight-Initialization on Restricted Boltzmann Machine
• A Probability Density Theory for Spin-Glass Systems
• Landscape Complexity for the Empirical Risk of Generalized Linear Models: Discrimination between Structured Data
• The Expressivity and Training of Deep Neural Networks: toward the Edge of Chaos?
• A novel derivation of the Marchenko-Pastur law through analog bipartite spin-glasses