Short-Term Memory in Orthogonal Neural Networks

Abstract

We study the ability of linear recurrent networks obeying discrete time dynamics to store long temporal sequences that are retrievable from the instantaneous state of the network. We calculate this temporal memory capacity for both distributed shift register and random orthogonal connectivity matrices. We show that the memory capacity of these networks scales with system size.

4 pages, 4 figures, to be published in Phys. Rev. Lett

Cited by in corpus (33)

• Physical reservoir computing -- An introductory perspective
• Learning universal computations with spikes
• The difference between memory and prediction in linear recurrent networks
• Optimal short-term memory before the edge of chaos in driven random recurrent networks
• Memory and forecasting capacities of nonlinear recurrent networks
• Model-Size Reduction for Reservoir Computing by Concatenating Internal States Through Time
• Binary embeddings with structured hashed projections
• Risk bounds for reservoir computing
• Neural mechanism to simulate a scale-invariant future timeline
• Stability Analysis of Reservoir Computers Dynamics via Lyapunov Functions
• Reservoir Topology in Deep Echo State Networks
• Nearly extensive sequential memory lifetime achieved by coupled nonlinear neurons
• Signal buffering in random networks of spiking neurons: microscopic vs. macroscopic phenomena
• The Computational Capacity of LRC, Memristive and Hybrid Reservoirs
• Time cells might be optimized for predictive capacity, not redundancy reduction or memory capacity
• Distributed Sequence Memory of Multidimensional Inputs in Recurrent Networks
• Universal discrete-time reservoir computers with stochastic inputs and linear readouts using non-homogeneous state-affine systems
• Product Reservoir Computing: Time-Series Computation with Multiplicative Neurons
• Differentiable reservoir computing
• Orthogonal Echo State Networks and stochastic evaluations of likelihoods
• Efficient Time-Series Approximation with Linear Recurrent Neural Networks: Architecture Learning and Predictive Power
• Context-dependent representation in recurrent neural networks
• New analytic formulae for memory and prediction functions in reservoir computers with time delays
• Intrinsic computation of a Monod-Wyman-Changeux molecule
• Exploring Transfer Function Nonlinearity in Echo State Networks
• Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks
• Learning recurrent dynamics in spiking networks
• Neural Population Coding of Multiple Stimuli
• On Simplicity and Complexity in the Brave New World of Large-Scale Neuroscience
• Analysis of Recurrent Linear Networks for Enabling Compressed Sensing of Time-Varying Signals
• Short Term Memory Capacity in Networks via the Restricted Isometry Property
• Optimally fuzzy temporal memory
• Transient Information Flow in a Network of Excitatory and Inhibitory Model Neurons: Role of Noise and Signal Autocorrelation