Mean-field message-passing equations in the Hopfield model and its generalizations

Abstract

Motivated by recent progress in using restricted Boltzmann machines as preprocessing algorithms for deep neural network, we revisit the mean-field equations (belief-propagation and TAP equations) in the best understood such machine, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polarizations of neurons. In the "retrieval phase" where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations : the set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations, or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines.

29 pages, 4 figures

References in corpus (4)

• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Hiding Quiet Solutions in Random Constraint Satisfaction Problems
• Efficient supervised learning in networks with binary synapses
• Advanced Mean Field Theory of Restricted Boltzmann Machine

Cited by in corpus (30)

• Restricted Boltzmann Machine, recent advances and mean-field theory
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Supervised Hebbian Learning
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• Quantum field-theoretic machine learning
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Quantifying Relevance in Learning and Inference
• Memory-free dynamics for the TAP equations of Ising models with arbitrary rotation invariant ensembles of random coupling matrices
• An inference problem in a mismatched setting: a spin-glass model with Mattis interaction
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Deep learning via message passing algorithms based on belief propagation
• Metastability of the Potts ferromagnet on random regular graphs
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
• Eigenvalue spectrum of neural networks with arbitrary Hebbian length
• Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines
• Inference and mutual information on random factor graphs
• A residual-based message passing algorithm for constraint satisfaction problems
• On the mapping between Hopfield networks and Restricted Boltzmann Machines
• Recognition Capabilities of a Hopfield Model with Auxiliary Hidden Neurons
• The cavity method for minority games between arbitrageurs on financial markets
• Statistical mechanics of vector Hopfield network near and above saturation
• On the role of non-linear latent features in bipartite generative neural networks
• Critical Dynamics and Cyclic Memory Retrieval in Non-reciprocal Hopfield Networks
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data
• Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings
• Topological constraints on self-organisation in locally interacting systems