Low-Dimensional Manifolds Support Multiplexed Integrations in Recurrent Neural Networks
arXiv:2011.10435
Abstract
We study the learning dynamics and the representations emerging in Recurrent Neural Networks trained to integrate one or multiple temporal signals. Combining analytical and numerical investigations, we characterize the conditions under which a RNN with n neurons learns to integrate D(n) scalar signals of arbitrary duration. We show, both for linear and ReLU neurons, that its internal state lives close to a D-dimensional manifold, whose shape is related to the activation function. Each neuron therefore carries, to various degrees, information about the value of all integrals. We discuss the deep analogy between our results and the concept of mixed selectivity forged by computational neuroscientists to interpret cortical recordings.
References in corpus (9)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Methods for Interpreting and Understanding Deep Neural Networks
- Recent Advances in Physical Reservoir Computing: A Review
- A Critical Review of Recurrent Neural Networks for Sequence Learning
- Understanding Neural Networks through Representation Erasure
- Linking connectivity, dynamics and computations in low-rank recurrent neural networks
- Learning to Generate Reviews and Discovering Sentiment
- The interplay between randomness and structure during learning in RNNs