Dense Hebbian neural networks: a replica symmetric picture of supervised learning

Abstract

We consider dense, associative neural-networks trained by a teacher (i.e., with supervision) and we investigate their computational capabilities analytically, via statistical-mechanics of spin glasses, and numerically, via Monte Carlo simulations. In particular, we obtain a phase diagram summarizing their performance as a function of the control parameters such as quality and quantity of the training dataset, network storage and noise, that is valid in the limit of large network size and structureless datasets: these networks may work in a ultra-storage regime (where they can handle a huge amount of patterns, if compared with shallow neural networks) or in a ultra-detection regime (where they can perform pattern recognition at prohibitive signal-to-noise ratios, if compared with shallow neural networks). Guided by the random theory as a reference framework, we also test numerically learning, storing and retrieval capabilities shown by these networks on structured datasets as MNist and Fashion MNist. As technical remarks, from the analytic side, we implement large deviations and stability analysis within Guerra's interpolation to tackle the not-Gaussian distributions involved in the post-synaptic potentials while, from the computational counterpart, we insert Plefka approximation in the Monte Carlo scheme, to speed up the evaluation of the synaptic tensors, overall obtaining a novel and broad approach to investigate supervised learning in neural networks, beyond the shallow limit, in general.

arXiv admin note: text overlap with arXiv:2211.14067

References in corpus (11)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
• Phase transition in the detection of modules in sparse networks
• Supervised Hebbian Learning
• Neural networks with redundant representation: detecting the undetectable
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Power-law Scaling to Assist with Key Challenges in Artificial Intelligence
• Replica symmetry breaking in dense neural networks
• Non-linear PDEs approach to statistical mechanics of Dense Associative Memories
• Large deviations of the free energy in the p-spin glass spherical model
• Phase diagram for the tap energy of the $p$-spin spherical mean field spin glass model

Cited by in corpus (4)

• The Exponential Capacity of Dense Associative Memories
• Generalized hetero-associative neural networks
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• The effect of priors on Learning with Restricted Boltzmann Machines