Replica symmetry breaking in dense neural networks

Abstract

Understanding the glassy nature of neural networks is pivotal both for theoretical and computational advances in Machine Learning and Theoretical Artificial Intelligence. Keeping the focus on dense associative Hebbian neural networks, the purpose of this paper is two-fold: at first we develop rigorous mathematical approaches to address properly a statistical mechanical picture of the phenomenon of {\em replica symmetry breaking} (RSB) in these networks, then -- deepening results stemmed via these routes -- we aim to inspect the {\em glassiness} that they hide. In particular, regarding the methodology, we provide two techniques: the former is an adaptation of the transport PDE to the case, while the latter is an extension of Guerra's interpolation breakthrough. Beyond coherence among the results, either in replica symmetric and in the one-step replica symmetry breaking level of description, we prove the Gardner's picture and we identify the maximal storage capacity by a ground-state analysis in the Baldi-Venkatesh high-storage regime. In the second part of the paper we investigate the glassy structure of these networks: in contrast with the replica symmetric scenario (RS), RSB actually stabilizes the spin-glass phase. We report huge differences w.r.t. the standard pairwise Hopfield limit: in particular, it is known that it is possible to express the free energy of the Hopfield neural network as a linear combination of the free energies of an hard spin glass (i.e. the Sherrington-Kirkpatrick model) and a soft spin glass (the Gaussian or "spherical" model). This is no longer true when interactions are more than pairwise (whatever the level of description, RS or RSB): for dense networks solely the free energy of the hard spin glass survives, proving a huge diversity in the underlying glassiness of associative neural networks.

References in corpus (23)

• Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
• The Parisi formula has a unique minimizer
• The free energy in a multi-species Sherrington-Kirkpatrick model
• Equilibrium statistical mechanics of bipartite spin systems
• Shaping the learning landscape in neural networks around wide flat minima
• Weight Space Structure and Internal Representations: a Direct Approach to Learning and Generalization in Multilayer Neural Network
• Dense Associative Memory is Robust to Adversarial Inputs
• Free energy and complexity of spherical bipartite models
• Replica Symmetry Breaking in Attractor Neural Network Models
• How glassy are neural networks?
• Replica Symmetry Breaking in Bipartite Spin Glasses and Neural Networks
• Replica symmetry breaking in mean field spin glasses trough Hamilton-Jacobi technique
• Neural networks with redundant representation: detecting the undetectable
• Universality in bipartite mean field spin glasses
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Dreaming neural networks: rigorous results
• Free energy in multi-species mixed $p$-spin spherical models
• Notes on the p-spin glass studied via Hamilton-Jacobi and Smooth-Cavity techniques
• Replica symmetry breaking in multi-species Sherrington-Kirkpatrick model
• Crisanti-Sommers formula and simultaneous symmetry breaking in multi-species spherical spin glasses
• Existence of two-step replica symmetry breaking for the spherical mixed p-spin glass at zero temperature
• PDE/statistical mechanics duality: relation between Guerra's interpolated $p$-spin ferromagnets and the Burgers hierarchy
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories

Cited by in corpus (4)

• Dense Hebbian neural networks: a replica symmetric picture of supervised learning
• Dense Hopfield Networks in the Teacher-Student Setting
• On the role of non-linear latent features in bipartite generative neural networks
• Supervised and Unsupervised protocols for hetero-associative neural networks