Replica symmetry breaking in dense neural networks

Abstract

Understanding the glassy nature of neural networks is pivotal both for theoretical and computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping the focus on dense associative Hebbian neural networks, the purpose of this paper is two-fold: at first we develop rigorous mathematical approaches to address properly a statistical mechanical picture of the phenomenon of {\em replica symmetry breaking} (RSB) in these networks, then -- deepening results stemmed via these routes -- we aim to inspect the {\em glassiness} that they hide. In particular, regarding the methodology, we provide two techniques: the former is an adaptation of the transport PDE to the case, while the latter is an extension of Guerra's interpolation breakthrough. Beyond coherence among the results, either in replica symmetric and in the one-step replica symmetry breaking level of description, we prove the Gardner's picture and we identify the maximal storage capacity by a ground-state analysis in the Baldi-Venkatesh high-storage regime. In the second part of the paper we investigate the glassy structure of these networks: in contrast with the replica symmetric scenario (RS), RSB actually stabilizes the spin-glass phase. We report huge differences w.r.t. the standard pairwise Hopfield limit: in particular, it is known that it is possible to express the free energy of the Hopfield neural network as a linear combination of the free energies of an hard spin glass (i.e. the Sherrington-Kirkpatrick model) and a soft spin glass (the Gaussian or "spherical" model). This is no longer true when interactions are more than pairwise (whatever the level of description, RS or RSB): for dense networks solely the free energy of the hard spin glass survives, proving a huge diversity in the underlying glassiness of associative neural networks.

References in corpus (7)

• Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
• Free energy and complexity of spherical bipartite models
• Neural networks with redundant representation: detecting the undetectable
• Free energy in multi-species mixed $p$-spin spherical models
• Crisanti-Sommers formula and simultaneous symmetry breaking in multi-species spherical spin glasses
• PDE/statistical mechanics duality: relation between Guerra's interpolated $p$-spin ferromagnets and the Burgers hierarchy
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories

Cited by in corpus (4)

• Dense Hebbian neural networks: a replica symmetric picture of supervised learning
• Dense Hopfield Networks in the Teacher-Student Setting
• On the role of non-linear latent features in bipartite generative neural networks
• Supervised and Unsupervised protocols for hetero-associative neural networks