Hierarchical neural networks perform both serial and parallel processing
arXiv:1409.0227 · doi:10.1016/j.neunet.2015.02.010
Abstract
In this work we study a Hebbian neural network, where neurons are arranged according to a hierarchical architecture such that their couplings scale with their reciprocal distance. As a full statistical mechanics solution is not yet available, after a streamlined introduction to the state of the art via that route, the problem is consistently approached through signal- to-noise technique and extensive numerical simulations. Focusing on the low-storage regime, where the amount of stored patterns grows at most logarithmical with the system size, we prove that these non-mean-field Hopfield-like networks display a richer phase diagram than their classical counterparts. In particular, these networks are able to perform serial processing (i.e. retrieve one pattern at a time through a complete rearrangement of the whole ensemble of neurons) as well as parallel processing (i.e. retrieve several patterns simultaneously, delegating the management of diff erent patterns to diverse communities that build network). The tune between the two regimes is given by the rate of the coupling decay and by the level of noise affecting the system. The price to pay for those remarkable capabilities lies in a network's capacity smaller than the mean field counterpart, thus yielding a new budget principle: the wider the multitasking capabilities, the lower the network load and viceversa. This may have important implications in our understanding of biological complexity.
References in corpus (22)
- Statistical mechanics of complex networks
- Broken Replica Symmetry Bounds in the Mean Field Spin Glass Model
- The Thermodynamic Limit in Mean Field Spin Glass Models
- Multitasking associative networks
- Equilibrium statistical mechanics of bipartite spin systems
- The replica symmetric behavior of the analogical neural network
- Extensive load in multitasking associative networks
- From Dyson to Hopfield: Processing on hierarchical networks
- Immune networks: multi-tasking capabilities near saturation
- The Hierarchical Random Energy Model
- How glassy are neural networks?
- Analytic solution of attractor neural networks on scale-free graphs
- A Hebbian approach to complex network generation
- Criticality in diluted ferromagnet
- Parallel processing in immune networks
- Meta-stable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network
- Notes on the Statistical Mechanics of Systems with Long-Range Interactions
- Transition between localized and extended states in the hierarchical Anderson model
- Scaling of the largest dynamical barrier in the one-dimensional long-range Ising spin-glass
- Free-energy bounds for hierarchical spin models
- The renormalization flow of the hierarchical Anderson model at weak disorder
- Dynamical Barriers in the Dyson Hierarchical model via Real Space Renormalization
Cited by in corpus (14)
- Hierarchical organization of functional connectivity in the mouse brain: a complex network approach
- Non-Convex Multi-species Hopfield models
- Solving Optimization Problems by the Public Goods Game
- First-passage phenomena in hierarchical networks
- Hopfield model with planted patterns: a teacher-student self-supervised learning model
- Topological properties of hierarchical networks
- Real-Space Renormalization Group for Spectral Properties of Hierarchical Networks
- Dense Hopfield Networks in the Teacher-Student Setting
- Modeling Structured Data Learning with Restricted Boltzmann Machines in the Teacher-Student Setting
- An Evolutionary Strategy based on Partial Imitation for Solving Optimization Problems
- The effect of priors on Learning with Restricted Boltzmann Machines
- Optimized brute-force algorithms for the bifurcation analysis of a spin-glass-like neural network model
- The dual of the space of interactions in neural network models
- A relativistic extension of Hopfield neural networks via the mechanical analogy