Neural Networks retrieving Boolean patterns in a sea of Gaussian ones

Abstract

Restricted Boltzmann Machines are key tools in Machine Learning and are described by the energy function of bipartite spin-glasses. From a statistical mechanical perspective, they share the same Gibbs measure of Hopfield networks for associative memory. In this equivalence, weights in the former play as patterns in the latter. As Boltzmann machines usually require real weights to be trained with gradient descent like methods, while Hopfield networks typically store binary patterns to be able to retrieve, the investigation of a mixed Hebbian network, equipped with both real (e.g., Gaussian) and discrete (e.g., Boolean) patterns naturally arises. We prove that, in the challenging regime of a high storage of real patterns, where retrieval is forbidden, an extra load of Boolean patterns can still be retrieved, as long as the ratio among the overall load and the network size does not exceed a critical threshold, that turns out to be the same of the standard Amit-Gutfreund-Sompolinsky theory. Assuming replica symmetry, we study the case of a low load of Boolean patterns combining the stochastic stability and Hamilton-Jacobi interpolating techniques. The result can be extended to the high load by a non rigorous but standard replica computation argument.

16 pages, 1 figure

References in corpus (9)

• Mean-field message-passing equations in the Hopfield model and its generalizations
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Extensive load in multitasking associative networks
• Phase transitions in Restricted Boltzmann Machines with generic priors
• The mean field Ising model trough interpolating techniques
• Free energy and complexity of spherical bipartite models
• Advanced Mean Field Theory of Restricted Boltzmann Machine
• Statistical mechanics of unsupervised feature learning in a restricted Boltzmann machine with binary synapses
• Unsupervised feature learning from finite data by message passing: discontinuous versus continuous phase transition

Cited by in corpus (11)

• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• Neural networks with redundant representation: detecting the undetectable
• Dreaming neural networks: rigorous results
• Dense Hebbian neural networks: a replica symmetric picture of supervised learning
• Replica symmetry breaking in dense neural networks
• Hopfield model with planted patterns: a teacher-student self-supervised learning model
• Dense Hopfield Networks in the Teacher-Student Setting
• The effect of priors on Learning with Restricted Boltzmann Machines
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
• The Relativistic Hopfield network: rigorous results
• A generalized Hopfield model to store and retrieve mismatched memory patterns