Statistical mechanics of vector Hopfield network near and above saturation

Abstract

We study analytically and numerically a Hopfield fully-connected network with $d$-dimensional vector spins. These networks are models of associative memory that generalize the standard Hopfield with Ising spins, where $P$ examples are stored in a network of $N$ units as local minima in an energy landscape. We study the equilibrium and out-of-equilibrium properties of the system, considering the system in its retrieval phase $\alpha <{\alpha }_c$ and beyond, where $\alpha =P/N$ is the capacity of the system and ${\alpha }_c$ is its critical value, above which storage fails. We derive the Replica Symmetric solution for the equilibrium thermodynamics of the system, together with its phase diagram: we find that the retrieval phase of the network shrinks with growing spin dimension, having ultimately a vanishing critical capacity ${\alpha }_c\propto 1/d$ in the large $d$ limit. As a trade-off, we observe that in the same limit vector Hopfield networks are able to denoise corrupted input patterns in the first step of retrieval dynamics, up to very large capacities $\alpha \propto d$. We also study the static properties of the system at zero temperature, considering the statistical properties of soft modes of the energy Hessian spectrum. We find that local minima of the energy landscape related to memory states have ungapped spectra with rare soft eigenmodes: these excitations are localized, their measure condensating on the noisiest neurons of the memory state.

44 pages, 13 figures. Revised version and new appendix: Statistical analysis of local effects: differential alignments and noisy spins

References in corpus (14)

• Mean-field message-passing equations in the Hopfield model and its generalizations
• Eigenvectors under a generic perturbation: non-perturbative results from the random matrix approach
• The Exponential Capacity of Dense Associative Memories
• Mean-field model of interacting quasilocalized excitations in glasses
• Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks
• The nature of non-phononic excitations in disordered systems
• Delocalization transition in low energy excitation modes of vector spin glasses
• Generalized hetero-associative neural networks
• Bose-Einstein-Like condensation of deformed random matrix: A replica approach
• Low energy excitations of mean-field glasses
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• Experimental evidence for the ${\omega }^4$ tail of the nonphononic spectra of glasses
• The Spectral Boundary of Block Structured Random Matrices
• Soft modes in vector spin glass models on sparse random graphs