Dense Hopfield Networks in the Teacher-Student Setting

Abstract

Dense Hopfield networks are known for their feature to prototype transition and adversarial robustness. However, previous theoretical studies have been mostly concerned with their storage capacity. We bridge this gap by studying the phase diagram of p-body Hopfield networks in the teacher-student setting of an unsupervised learning problem, uncovering ferromagnetic phases reminiscent of the prototype and feature learning regimes. On the Nishimori line, we find the critical size of the training set necessary for efficient pattern retrieval. Interestingly, we find that that the paramagnetic to ferromagnetic transition of the teacher-student setting coincides with the paramagnetic to spin-glass transition of the direct model, i.e. with random patterns. Outside of the Nishimori line, we investigate the learning performance in relation to the inference temperature and dataset noise. Moreover, we show that using a larger p for the student than the teacher gives the student an extensive tolerance to noise. We then derive a closed-form expression measuring the adversarial robustness of such a student at zero temperature, corroborating the positive correlation between number of parameters and robustness observed in large neural networks. We also use our model to clarify why the prototype phase of modern Hopfield networks is adversarially robust.

34 pages, 9 figures, updated to match published version, implemented minor changes proposed in referee reports

References in corpus (39)

• Evasion Attacks against Machine Learning at Test Time
• The structural glass transition and the entropy of the metastable states
• Statistical physics of inference: Thresholds and algorithms
• Phase Transitions in the Coloring of Random Graphs
• On the nature of the low-temperature phase in discontinuous mean-field spin glasses
• Algorithmic barriers from phase transitions
• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Hiding Quiet Solutions in Random Constraint Satisfaction Problems
• Multitasking associative networks
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• The Nishimori line and Bayesian Statistics
• On a model of associative memory with huge storage capacity
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Hopfield Networks is All You Need
• Dense Associative Memory is Robust to Adversarial Inputs
• Extensive load in multitasking associative networks
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Large Associative Memory Problem in Neurobiology and Machine Learning
• From Dyson to Hopfield: Processing on hierarchical networks
• Immune networks: multi-tasking capabilities near saturation
• Complexity in Mean-Field Spin-Glass Models: Ising $p$-spin
• Quiet Planting in the Locked Constraint Satisfaction Problems
• How glassy are neural networks?
• The Exponential Capacity of Dense Associative Memories
• Neural networks with redundant representation: detecting the undetectable
• Non-Convex Multi-species Hopfield models
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Hierarchical neural networks perform both serial and parallel processing
• Replica symmetry breaking in dense neural networks
• Minimal model of permutation symmetry in unsupervised learning
• Meta-stable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Legendre Equivalences of Spherical Boltzmann Machines
• Topological properties of hierarchical networks
• Overparameterized Linear Regression under Adversarial Attacks
• A law of robustness for two-layers neural networks
• From the replica trick to the replica symmetry breaking technique
• Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models
• A Waddington landscape for prototype learning in generalized Hopfield networks

Cited by in corpus (6)

• The effect of priors on Learning with Restricted Boltzmann Machines
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• Exact Replica Symmetric solution for transverse field Hopfield model under finite Trotter size
• Saddle Hierarchy in Dense Associative Memory
• Supervised and Unsupervised protocols for hetero-associative neural networks
• Multiphoton quantum simulation of the generalized Hopfield memory model