Dense Hebbian neural networks: a replica symmetric picture of supervised learning

Abstract

We consider dense, associative neural-networks trained by a teacher (i.e., with supervision) and we investigate their computational capabilities analytically, via statistical-mechanics of spin glasses, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as quality and quantity of the training dataset, network storage and noise, that is valid in the limit of large network size and structureless datasets: these networks may work in a ultra-storage regime (where they can handle a huge amount of patterns, if compared with shallow neural networks) or in a ultra-detection regime (where they can perform pattern recognition at prohibitive signal-to-noise ratios, if compared with shallow neural networks). Guided by the random theory as a reference framework, we also test numerically learning, storing and retrieval capabilities shown by these networks on structured datasets as MNist and Fashion MNist. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate supervised learning in neural networks, beyond the shallow limit, in general.

arXiv commentarXiv admin note: text overlap with arXiv:2211.14067

References in corpus (22)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
• Spin-Glass Theory for Pedestrians
• Phase transition in the detection of modules in sparse networks
• The replica symmetric behavior of the analogical neural network
• Dense Associative Memory is Robust to Adversarial Inputs
• Large Associative Memory Problem in Neurobiology and Machine Learning
• Supervised Hebbian Learning
• How glassy are neural networks?
• Neural networks with redundant representation: detecting the undetectable
• Universality in bipartite mean field spin glasses
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Mechanisms of dimensionality reduction and decorrelation in deep neural networks
• Power-law Scaling to Assist with Key Challenges in Artificial Intelligence
• Notes on the p-spin glass studied via Hamilton-Jacobi and Smooth-Cavity techniques
• Thouless-Anderson-Palmer equations for the generic p-spin glass model
• Replica symmetry breaking in dense neural networks
• Extended Plefka Expansion for Stochastic Dynamics
• Non-linear PDEs approach to statistical mechanics of Dense Associative Memories
• Large deviations of the free energy in the p-spin glass spherical model
• Phase diagram for the tap energy of the $p$-spin spherical mean field spin glass model

Cited by in corpus (4)

• The Exponential Capacity of Dense Associative Memories
• Generalized hetero-associative neural networks
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• The effect of priors on Learning with Restricted Boltzmann Machines