Efficiency of local learning rules in threshold-linear associative networks
arXiv:2007.12584 · doi:10.1103/PhysRevLett.126.018301
Abstract
We derive the Gardner storage capacity for associative networks of threshold linear units, and show that with Hebbian learning they can operate closer to such Gardner bound than binary networks, and even surpass it. This is largely achieved through a sparsification of the retrieved patterns, which we analyze for theoretical and empirical distributions of activity. As reaching the optimal capacity via non-local learning rules like backpropagation requires slow and neurally implausible training procedures, our results indicate that one-shot self-organized Hebbian learning can be just as efficient.
References in corpus (5)
- Unsupervised Learning by Competing Hidden Units
- A balanced memory network
- Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
- Localized activity profiles and storage capacity of rate-based autoassociative networks
- Disappearance of Spurious States in Analog Associative Memories