Hopfield model with planted patterns: a teacher-student self-supervised learning model

Abstract

While Hopfield networks are known as paradigmatic models for memory storage and retrieval, modern artificial intelligence systems mainly stand on the machine learning paradigm. We show that it is possible to formulate a teacher-student self-supervised learning problem with Boltzmann machines in terms of a suitable generalization of the Hopfield model with structured patterns, where the spin variables are the machine weights and patterns correspond to the training set's examples. We analyze the learning performance by studying the phase diagram in terms of the training set size, the dataset noise and the inference temperature (i.e. the weight regularization). With a small but informative dataset the machine can learn by memorization. With a noisy dataset, an extensive number of examples above a critical threshold is needed. In this regime the memory storage limits of the system becomes an opportunity for the occurrence of a learning regime in which the system can generalize.

arXiv comment27 pages, 5 figures, typo in the free energy corrected

References in corpus (36)

• Multitasking associative networks
• The Nishimori line and Bayesian Statistics
• Mean-field message-passing equations in the Hopfield model and its generalizations
• The free energy in a multi-species Sherrington-Kirkpatrick model
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Extensive load in multitasking associative networks
• From Dyson to Hopfield: Processing on hierarchical networks
• Immune networks: multi-tasking capabilities near saturation
• Supervised Hebbian Learning
• High-Dimensional Inference with the generalized Hopfield Model: Principal Component Analysis and Corrections
• How glassy are neural networks?
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Non-Convex Multi-species Hopfield models
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Deep Boltzmann machines: rigorous results at arbitrary depth
• Limits and performances of algorithms based on simulated annealing in solving sparse hard inference problems
• Hierarchical neural networks perform both serial and parallel processing
• Mean field bipartite spin models treated with mechanical techniques
• The solution of the deep Boltzmann machine on the Nishimori line
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
• Solving the inverse Ising problem by mean-field methods in a clustered phase space with many states
• The multi-species mean-field spin-glass on the Nishimori line
• Inference and learning in sparse systems with multiple states
• Minimal model of permutation symmetry in unsupervised learning
• Meta-stable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network
• Legendre Duality of Spherical and Gaussian Spin Glasses
• Legendre Equivalences of Spherical Boltzmann Machines
• Topological properties of hierarchical networks
• Non-Convex Multipartite Ferromagnets
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Overlap synchronisation in multipartite random energy models
• Effect of spatial correlations on Hopfield Neural Network and Dense Associative Memories
• A remark on the spherical bipartite spin glass
• Mismatching as a tool to enhance algorithmic performances of Monte Carlo methods for the planted clique model

Cited by in corpus (6)

• Dense Hopfield Networks in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• On the role of non-linear latent features in bipartite generative neural networks
• Saddle Hierarchy in Dense Associative Memory
• Supervised and Unsupervised protocols for hetero-associative neural networks