Neural networks with redundant representation: detecting the undetectable

Abstract

We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order P=4. The latter is known to be able to Hebbian-store an amount of patterns scaling as N^{P-1}, where N denotes the number of constituting binary neurons interacting P-wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with N, while P>2) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with P=4 is able to retrieve information whose intensity is O(1) even in the presence of a noise O(\sqrt{N}) in the large N limit. This striking skill stems from a redundancy representation of patterns -- which is afforded given the (relatively) low-load information storage -- and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.

References in corpus (13)

• Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
• An exact mapping between the Variational Renormalization Group and Deep Learning
• Small-correlation expansions for the inverse Ising problem
• Equilibrium statistical mechanics of bipartite spin systems
• Phase Diagram of Restricted Boltzmann Machines and Generalised Hopfield Networks with Arbitrary Priors
• The replica symmetric behavior of the analogical neural network
• Dense Associative Memory is Robust to Adversarial Inputs
• The mean field Ising model trough interpolating techniques
• Irreducible free energy expansion and overlaps locking in mean field spin glasses
• Universality in bipartite mean field spin glasses
• Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
• Resummed mean-field inference for strongly coupled data
• The spin-glass phase-transition in the Hopfield model with p-spin interactions

Cited by in corpus (15)

• Homophily based on few attributes can impede structural balance
• The Exponential Capacity of Dense Associative Memories
• Dense Hebbian neural networks: a replica symmetric picture of supervised learning
• An inference problem in a mismatched setting: a spin-glass model with Mattis interaction
• Replica symmetry breaking in dense neural networks
• Differentiable Physics: A Position Piece
• Deep learning of deformation-dependent conductance in thin films: nanobubbles in graphene
• Non-linear PDEs approach to statistical mechanics of Dense Associative Memories
• Dense Hopfield Networks in the Teacher-Student Setting
• PDE/statistical mechanics duality: relation between Guerra's interpolated $p$-spin ferromagnets and the Burgers hierarchy
• Interpolating between boolean and extremely high noisy patterns through Minimal Dense Associative Memories
• Dense Associative Memory in a Nonlinear Optical Hopfield Neural Network
• Rigorous approaches for spin glass and Gaussian spin glass with P-wise interactions
• Supervised and Unsupervised protocols for hetero-associative neural networks
• Multiphoton quantum simulation of the generalized Hopfield memory model