Mean-field theory for a spin-glass model of neural networks: TAP free energy and paramagnetic to spin-glass transition

Abstract

An approach is proposed to the Hopfield model where the mean-field treatment is made for a given set of stored patterns (sample) and then the statistical average over samples is taken. This corresponds to the approach made by Thouless, Anderson and Palmer (TAP) to the infinite-range model of spin glasses. Taking into account the fact that in the Hopfield model there exist correlations between different elements of the interaction matrix, we obtain its TAP free energy explicitly, which consists of a series of terms exhibiting the cluster effect. Nature of the spin-glass transition in the model is also examined and compared with those given by the replica method as well as the cavity method.

12 pages, LaTex, 1 PostScript figure

Cited by in corpus (16)

• The Loss Surfaces of Multilayer Networks
• Cavity Approach to the Spectral Density of Sparse Symmetric Random Matrices
• Mean-field message-passing equations in the Hopfield model and its generalizations
• High-temperature Expansions and Message Passing Algorithms
• Associative memory storing an extensive number of patterns based on a network of oscillators with distributed natural frequencies in the presence of external white noise
• The Wishart planted ensemble: A tunably-rugged pairwise Ising model with a first-order phase transition
• Correlated fluctuations in strongly-coupled binary networks beyond equilibrium
• Higher order corrections to the effective potential close to the jamming transition in the perceptron model
• Expansion of the Gibbs potential for quantum many-body systems: General formalism with applications to the spin glass and the weakly non-ideal Bose gas
• Expansion of the effective action around non-Gaussian theories
• Eigenvalue spectrum of neural networks with arbitrary Hebbian length
• Black holes and the loss landscape in machine learning
• Ising distribution as a latent variable model
• Dynamical mean field approach to associative memory model with non-monotonic transfer functions
• Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks
• Diagrammatics of free energies with fixed variance for high-dimensional data