The Exponential Capacity of Dense Associative Memories

Abstract

Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp (\alpha N)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, ${\alpha }_1$, and lower bounds for the maximum of the load $\alpha$ for which all patterns can be retrieved, ${\alpha }_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.

Version accepted on Physics Review Letters

References in corpus (2)

• Neural networks with redundant representation: detecting the undetectable
• Dense Hebbian neural networks: a replica symmetric picture of supervised learning

Cited by in corpus (13)

• Dynamical Regimes of Diffusion Models
• Controlled asymmetric Ising model implemented with parametric micromechanical oscillators
• The unbearable lightness of Restricted Boltzmann Machines: Theoretical Insights and Biological Applications
• Dense Hopfield Networks in the Teacher-Student Setting
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• Explosive neural networks via higher-order interactions in curved statistical manifolds
• Nadaraya-Watson kernel smoothing as a random energy model
• Analysis of Diffusion Models for Manifold Data
• Dense Associative Memory in a Nonlinear Optical Hopfield Neural Network
• A simplified Parisi Ansatz II: REM universality
• Statistical mechanics of vector Hopfield network near and above saturation
• Saddle Hierarchy in Dense Associative Memory
• Dynamical Regimes of Discrete Diffusion Models