Non-Convex Multi-species Hopfield models

Abstract

In this work we introduce a multi-species generalization of the Hopfield model for associative memory, where neurons are divided into groups and both inter-groups and intra-groups pair-wise interactions are considered, with different intensities. Thus, this system contains two of the main ingredients of modern Deep neural network architectures: Hebbian interactions to store patterns of information and multiple layers coding different levels of correlations. The model is completely solvable in the low-load regime with a suitable generalization of the Hamilton-Jacobi technique, despite the Hamiltonian can be a non-definite quadratic form of the magnetizations. The family of multi-species Hopfield model includes, as special cases, the 3-layers Restricted Boltzmann Machine (RBM) with Gaussian hidden layer and the Bidirectional Associative Memory (BAM) model.

arXiv commentThis is a pre-print of an article published in J. Stat. Phys

References in corpus (28)

• Multitasking associative networks
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Phase transitions and sample complexity in Bayes-optimal matrix factorization
• Mean-field message-passing equations in the Hopfield model and its generalizations
• The free energy in a multi-species Sherrington-Kirkpatrick model
• Equilibrium statistical mechanics of bipartite spin systems
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• The replica symmetric behavior of the analogical neural network
• Finite Connectivity Attractor Neural Networks
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Extensive load in multitasking associative networks
• The mean field Ising model trough interpolating techniques
• From Dyson to Hopfield: Processing on hierarchical networks
• Immune networks: multi-tasking capabilities near saturation
• A mechanical approach to mean field spin models
• How glassy are neural networks?
• Replica symmetry breaking in mean field spin glasses trough Hamilton-Jacobi technique
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Hierarchical neural networks perform both serial and parallel processing
• Mean field bipartite spin models treated with mechanical techniques
• Inverse problem robustness for multi-species mean field spin models
• Meta-stable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network
• Legendre Duality of Spherical and Gaussian Spin Glasses
• Topological properties of hierarchical networks
• Non-Convex Multipartite Ferromagnets
• High storage capacity in the Hopfield model with auto-interactions - stability analysis
• Role of zero synapses in unsupervised feature learning
• Overlap synchronisation in multipartite random energy models

Cited by in corpus (11)

• Quantum field-theoretic machine learning
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Generalized hetero-associative neural networks
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• Non-linear PDEs approach to statistical mechanics of Dense Associative Memories
• Dense Hopfield Networks in the Teacher-Student Setting
• Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• The Relativistic Hopfield network: rigorous results
• Supervised and Unsupervised protocols for hetero-associative neural networks