Dreaming neural networks: rigorous results

Abstract

Recently a daily routine for associative neural networks has been proposed: the network Hebbian-learns during the awake state (thus behaving as a standard Hopfield model), then, during its sleep state, optimizing information storage, it consolidates pure patterns and removes spurious ones: this forces the synaptic matrix to collapse to the projector one (ultimately approaching the Kanter-Sompolinksy model). This procedure keeps the learning Hebbian-based (a biological must) but, by taking advantage of a (properly stylized) sleep phase, still reaches the maximal critical capacity (for symmetric interactions). So far this emerging picture (as well as the bulk of papers on unlearning techniques) was supported solely by mathematically-challenging routes, e.g. mainly replica-trick analysis and numerical simulations: here we rely extensively on Guerra's interpolation techniques developed for neural networks and, in particular, we extend the generalized stochastic stability approach to the case. Confining our description within the replica symmetric approximation (where the previous ones lie), the picture painted regarding this generalization (and the previously existing variations on theme) is here entirely confirmed. Further, still relying on Guerra's schemes, we develop a systematic fluctuation analysis to check where ergodicity is broken (an analysis entirely absent in previous investigations). We find that, as long as the network is awake, ergodicity is bounded by the Amit-Gutfreund-Sompolinsky critical line (as it should), but, as the network sleeps, sleeping destroys spin glass states by extending both the retrieval as well as the ergodic region: after an entire sleeping session the solely surviving regions are retrieval and ergodic ones and this allows the network to achieve the perfect retrieval regime (the number of storable patterns equals the number of neurons in the network).

References in corpus (14)

• Gibbs States and the Set of Solutions of Random Constraint Satisfaction Problems
• Statistical physics-based reconstruction in compressed sensing
• Adaptive Cluster Expansion for Inferring Boltzmann Machines with Noisy Data
• On the Stability of the Quenched State in Mean Field Spin Glass Models
• Multitasking associative networks
• Emergence of Compositional Representations in Restricted Boltzmann Machines
• Efficient supervised learning in networks with binary synapses
• Equilibrium statistical mechanics of bipartite spin systems
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• The replica symmetric behavior of the analogical neural network
• Phase transitions in Restricted Boltzmann Machines with generic priors
• Spin-Glass Stochastic Stability: a Rigorous Proof
• Universality in bipartite mean field spin glasses
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones

Cited by in corpus (9)

• Imaginary components of out-of-time correlators and information scrambling for navigating the learning landscape of a quantum machine learning model
• Replica symmetry breaking in dense neural networks
• Daydreaming Hopfield Networks and their surprising effectiveness on correlated data
• PDE/statistical mechanics duality: relation between Guerra's interpolated $p$-spin ferromagnets and the Burgers hierarchy
• The relativistic Hopfield model with correlated patterns
• On the role of non-linear latent features in bipartite generative neural networks
• A spectral approach to Hebbian-like neural networks
• Interacting Dreaming Neural Networks
• Daydreaming algorithm for Biased Patterns