Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
arXiv:1703.05210 · doi:10.1007/s10955-017-1840-9
Abstract
Restricted Boltzmann Machines are key tools in Machine Learning and are described by the energy function of bipartite spin-glasses. From a statistical mechanical perspective, they share the same Gibbs measure of Hopfield networks for associative memory. In this equivalence, weights in the former play as patterns in the latter. As Boltzmann machines usually require real weights to be trained with gradient descent like methods, while Hopfield networks typically store binary patterns to be able to retrieve, the investigation of a mixed Hebbian network, equipped with both real (e.g., Gaussian) and discrete (e.g., Boolean) patterns naturally arises. We prove that, in the challenging regime of a high storage of real patterns, where retrieval is forbidden, an extra load of Boolean patterns can still be retrieved, as long as the ratio among the overall load and the network size does not exceed a critical threshold, that turns out to be the same of the standard Amit-Gutfreund-Sompolinsky theory. Assuming replica symmetry, we study the case of a low load of Boolean patterns combining the stochastic stability and Hamilton-Jacobi interpolating techniques. The result can be extended to the high load by a non rigorous but standard replica computation argument.
arXiv comment16 pages, 1 figure
References in corpus (22)
- Searching for Exotic Particles in High-Energy Physics with Deep Learning
- On the Stability of the Quenched State in Mean Field Spin Glass Models
- Multitasking associative networks
- Emergence of Compositional Representations in Restricted Boltzmann Machines
- Mean-field message-passing equations in the Hopfield model and its generalizations
- Equilibrium statistical mechanics of bipartite spin systems
- Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
- The replica symmetric behavior of the analogical neural network
- Extensive load in multitasking associative networks
- Phase transitions in Restricted Boltzmann Machines with generic priors
- The mean field Ising model trough interpolating techniques
- From Dyson to Hopfield: Processing on hierarchical networks
- Free energy and complexity of spherical bipartite models
- Advanced Mean Field Theory of Restricted Boltzmann Machine
- Replica symmetry breaking in mean field spin glasses trough Hamilton-Jacobi technique
- Universality in bipartite mean field spin glasses
- Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
- Stochastic symmetry-breaking in a gaussian Hopfield model
- Mean field bipartite spin models treated with mechanical techniques
- Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition
- Interpolating the Sherrington-Kirkpatrick replica trick
- Non-Convex Multipartite Ferromagnets
Cited by in corpus (11)
- Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
- Neural networks with redundant representation: detecting the undetectable
- Dreaming neural networks: rigorous results
- Dense Hebbian neural networks: a replica symmetric picture of supervised learning
- Replica symmetry breaking in dense neural networks
- Hopfield model with planted patterns: a teacher-student self-supervised learning model
- Dense Hopfield Networks in the Teacher-Student Setting
- The effect of priors on Learning with Restricted Boltzmann Machines
- Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
- The Relativistic Hopfield network: rigorous results
- A generalized Hopfield model to store and retrieve mismatched memory patterns