Large Associative Memory Problem in Neurobiology and Machine Learning

Abstract

Dense Associative Memories or modern Hopfield networks permit storage and reliable retrieval of an exponentially large (in the dimension of feature space) number of memories. At the same time, their naive implementation is non-biological, since it seemingly requires the existence of many-body synaptic junctions between the neurons. We show that these models are effective descriptions of a more microscopic (written in terms of biological degrees of freedom) theory that has additional (hidden) neurons and only requires two-body interactions between them. For this reason our proposed microscopic theory is a valid model of large associative memory with a degree of biological plausibility. The dynamics of our network and its reduced dimensional equivalent both minimize energy (Lyapunov) functions. When certain dynamical variables (hidden neurons) are integrated out from our microscopic theory, one can recover many of the models that were previously discussed in the literature, e.g. the model presented in "Hopfield Networks is All You Need" paper. We also provide an alternative derivation of the energy function and the update rule proposed in the aforementioned paper and clarify the relationships between various models of this class.

Accepted for publication at ICLR 2021

References in corpus (2)

• Hopfield Networks is All You Need
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes

Cited by in corpus (14)

• Hopfield Networks is All You Need
• Enhancing associative memory recall and storage capacity using confocal cavity QED
• Theoretical foundations of studying criticality in the brain
• Dense Hebbian neural networks: a replica symmetric picture of supervised learning
• Hierarchical Associative Memory
• Eight challenges in developing theory of intelligence
• Unification of Symmetries Inside Neural Networks: Transformer, Feedforward and Neural ODE
• Dense Hopfield Networks in the Teacher-Student Setting
• Emergence of Human-Like Attention in Self-Supervised Vision Transformers: an eye-tracking study
• Dissipative Quantum Hopfield Network: A numerical analysis
• Unitary Evolutions Sourced By Interacting Quantum Memories: Closed Quantum Systems Directing Themselves Using Their State Histories
• Computational models of learning and synaptic plasticity
• Simplified derivations for high-dimensional convex learning problems
• Supervised and Unsupervised protocols for hetero-associative neural networks