Capacity-resolution trade-off in the optimal learning of multiple low-dimensional manifolds by attractor neural networks

Abstract

Recurrent neural networks (RNN) are powerful tools to explain how attractors may emerge from noisy, high-dimensional dynamics. We study here how to learn the ~N^(2) pairwise interactions in a RNN with N neurons to embed L manifolds of dimension D << N. We show that the capacity, i.e. the maximal ratio L/N, decreases as |log(epsilon)|^(-D), where epsilon is the error on the position encoded by the neural activity along each manifold. Hence, RNN are flexible memory devices capable of storing a large number of manifolds at high spatial resolution. Our results rely on a combination of analytical tools from statistical mechanics and random matrix theory, extending Gardner's classical theory of learning to the case of patterns with strong spatial correlations.

References in corpus (10)

• Scikit-learn: Machine Learning in Python
• CVXPY: A Python-Embedded Modeling Language for Convex Optimization
• Attractor neural networks storing multiple space representations: a model for hippocampal place fields
• Introduction to Random Matrices - Theory and Practice
• Classification and Geometry of General Perceptual Manifolds
• A Moving Bump in a Continuous Manifold: A Comprehensive Study of the Tracking Dynamics of Continuous Attractor Neural Networks
• Transitions between spatial attractors in place-cell networks
• Crosstalk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: Collective motion of the activity (II)
• Cross-talk and transitions between multiple spatial maps in an attractor neural network model of the hippocampus: phase diagram (I)
• On the Spectrum of Multi-Space Euclidean Random Matrices

Cited by in corpus (7)

• Supervised perceptron learning vs unsupervised Hebbian unlearning: Approaching optimal memory retrieval in Hopfield-like networks
• Linear Classification of Neural Manifolds with Correlated Variability
• 'Place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space
• Quantum Jamming: Critical Properties of a Quantum Mechanical Perceptron
• Associative memory model with arbitrary Hebbian length
• Optimal Learning with Excitatory and Inhibitory synapses
• On the Spectrum of Multi-Space Euclidean Random Matrices