Mean-field message-passing equations in the Hopfield model and its generalizations

Abstract

Motivated by recent progress in using restricted Boltzmann machines as preprocessing algorithms for deep neural network, we revisit the mean-field equations (belief-propagation and TAP equations) in the best understood such machine, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polarizations of neurons. In the "retrieval phase" where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations : the set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations, or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines.

arXiv comment29 pages, 4 figures

References in corpus (15)

• Message Passing Algorithms for Compressed Sensing
• The dynamics of message passing on dense graphs, with applications to compressed sensing
• Statistical physics of inference: Thresholds and algorithms
• Probabilistic Reconstruction in Compressed Sensing: Algorithms, Phase Diagrams, and Threshold Achieving Matrices
• Statistical physics-based reconstruction in compressed sensing
• Learning by message-passing in networks of discrete synapses
• Hiding Quiet Solutions in Random Constraint Satisfaction Problems
• Efficient supervised learning in networks with binary synapses
• MMSE of probabilistic low-rank matrix estimation: Universality with respect to the output channel
• Sparse PCA via Covariance Thresholding
• Mean-field theory for a spin-glass model of neural networks: TAP free energy and paramagnetic to spin-glass transition
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Inferring Sparsity: Compressed Sensing using Generalized Restricted Boltzmann Machines
• Estimation with Random Linear Mixing, Belief Propagation and Compressed Sensing
• A Max-Sum algorithm for training discrete neural networks

Cited by in corpus (59)

• Machine learning and the physical sciences
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Constrained Low-rank Matrix Estimation: Phase Transitions, Approximate Message Passing and Applications
• Generalisation error in learning with random features and the hidden manifold model
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Restricted Boltzmann Machine, recent advances and mean-field theory
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
• Supervised Hebbian Learning
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Spectral Dynamics of Learning Restricted Boltzmann Machines
• High-temperature Expansions and Message Passing Algorithms
• Quantum field-theoretic machine learning
• Mean-field inference methods for neural networks
• Annealing and replica-symmetry in Deep Boltzmann Machines
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Non-Convex Multi-species Hopfield models
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Statistical learning theory of structured data
• Memory-free dynamics for the TAP equations of Ising models with arbitrary rotation invariant ensembles of random coupling matrices
• Quantifying Relevance in Learning and Inference
• Approximate Survey Propagation for Statistical Inference
• A Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
• An inference problem in a mismatched setting: a spin-glass model with Mattis interaction
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Counting the learnable functions of structured data
• Minimal model of permutation symmetry in unsupervised learning
• Fundamental problems in statistical physics XIV: Lecture on Machine Learning
• Beyond the storage capacity: data driven satisfiability transition
• Spin systems on Bethe lattices
• Deep learning via message passing algorithms based on belief propagation
• Metastability of the Potts ferromagnet on random regular graphs
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Generalization from correlated sets of patterns in the perceptron
• Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
• The realizable solutions of the TAP equations
• High storage capacity in the Hopfield model with auto-interactions - stability analysis
• Gaussian-Spherical Restricted Boltzmann Machines
• Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
• Space of Functions Computed by Deep-Layered Machines
• Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisation
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
• Overlap synchronisation in multipartite random energy models
• Eigenvalue spectrum of neural networks with arbitrary Hebbian length
• Robust Multi-Output Learning with Highly Incomplete Data via Restricted Boltzmann Machines
• A residual-based message passing algorithm for constraint satisfaction problems
• Inference and mutual information on random factor graphs
• Recognition Capabilities of a Hopfield Model with Auxiliary Hidden Neurons
• On the mapping between Hopfield networks and Restricted Boltzmann Machines
• The cavity method for minority games between arbitrageurs on financial markets
• Generalized Approximate Survey Propagation for High-Dimensional Estimation
• On the role of non-linear latent features in bipartite generative neural networks
• Statistical mechanics of vector Hopfield network near and above saturation
• Critical Dynamics and Cyclic Memory Retrieval in Non-reciprocal Hopfield Networks
• Maximal Relevance and Optimal Learning Machines
• Exact training of Restricted Boltzmann machines on intrinsically low dimensional data
• Topological constraints on self-organisation in locally interacting systems
• Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings