Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
arXiv:1810.11075 · doi:10.1088/1742-5468/ab02ef
Abstract
Restricted Boltzmann machines (RBMs) constitute one of the main models for machine statistical inference and they are widely employed in Artificial Intelligence as powerful tools for (deep) learning. However, in contrast with countless remarkable practical successes, their mathematical formalization has been largely elusive: from a statistical-mechanics perspective these systems display the same (random) Gibbs measure of bi-partite spin-glasses, whose rigorous treatment is notoriously difficult. In this work, beyond providing a brief review on RBMs from both the learning and the retrieval perspectives, we aim to contribute to their analytical investigation, by considering two distinct realizations of their weights (i.e., Boolean and Gaussian) and studying the properties of their related free energies. More precisely, focusing on a RBM characterized by digital couplings, we first extend the Pastur-Shcherbina-Tirozzi method (originally developed for the Hopfield model) to prove the self-averaging property for the free energy, over its quenched expectation, in the infinite volume limit, then we explicitly calculate its simplest approximation, namely its annealed bound. Next, focusing on a RBM characterized by analogical weights, we extend Guerra's interpolating scheme to obtain a control of the quenched free-energy under the assumption of replica symmetry: we get self-consistencies for the order parameters (in full agreement with the existing Literature) as well as the critical line for ergodicity breaking that turns out to be the same obtained in AGS theory. As we discuss, this analogy stems from the slow-noise universality. Finally, glancing beyond replica symmetry, we analyze the fluctuations of the overlaps for an estimate of the (slow) noise affecting the retrieval of the signal, and by a stability analysis we recover the Aizenman-Contucci identities typical of glassy systems.
arXiv comment21 pages, 1 figure
References in corpus (25)
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
- Phase transition in the detection of modules in sparse networks
- The Thermodynamic Limit in Mean Field Spin Glass Models
- General properties of overlap probability distributions in disordered spin systems. Toward Parisi ultrametricity
- On the Stability of the Quenched State in Mean Field Spin Glass Models
- Multitasking associative networks
- Emergence of Compositional Representations in Restricted Boltzmann Machines
- Mean-field message-passing equations in the Hopfield model and its generalizations
- Equilibrium statistical mechanics of bipartite spin systems
- Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
- The replica symmetric behavior of the analogical neural network
- Phase transitions in Restricted Boltzmann Machines with generic priors
- Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
- Spectral Dynamics of Learning Restricted Boltzmann Machines
- A thermodynamical perspective of immune capabilities
- How glassy are neural networks?
- Advanced Mean Field Theory of Restricted Boltzmann Machine
- Universality in bipartite mean field spin glasses
- Non-Convex Multi-species Hopfield models
- Approximate Message Passing with Restricted Boltzmann Machine Priors
- Interpolating the Sherrington-Kirkpatrick replica trick
- Reconstructing the Hopfield network as an inverse Ising problem
- A dynamical model of the adaptive immune system: effects of cells promiscuity, antigens and B-B interactions
- The role of idiotypic interactions in the adaptive immune system: a belief-propagation approach
Cited by in corpus (5)
- Restricted Boltzmann Machine, recent advances and mean-field theory
- Non-linear PDEs approach to statistical mechanics of Dense Associative Memories
- The relativistic Hopfield model with correlated patterns
- Replica analysis of the lattice-gas Restricted Boltzmann Machine partition function
- Neuromodulation via Krotov-Hopfield Improves Accuracy and Robustness of RBMs