Memory and forecasting capacities of nonlinear recurrent networks

Abstract

The notion of memory capacity, originally introduced for echo state and linear networks with independent inputs, is generalized to nonlinear recurrent networks with stationary but dependent inputs. The presence of dependence in the inputs makes natural the introduction of the network forecasting capacity, that measures the possibility of forecasting time series values using network states. Generic bounds for memory and forecasting capacities are formulated in terms of the number of neurons of the nonlinear recurrent network and the autocovariance function or the spectral density of the input. These bounds generalize well-known estimates in the literature to a dependent inputs setup. Finally, for the particular case of linear recurrent networks with independent inputs it is proved that the memory capacity is given by the rank of the associated controllability matrix, a fact that has been for a long time assumed to be true without proof by the community.

arXiv comment27 pages, 1 figure. To appear in Physica D

References in corpus (8)

• Short-Term Memory in Orthogonal Neural Networks
• Determination of the edge of criticality in echo state networks through Fisher information maximization
• The difference between memory and prediction in linear recurrent networks
• Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere
• Risk bounds for reservoir computing
• Memory and Information Processing in Recurrent Neural Networks
• Distributed Sequence Memory of Multidimensional Inputs in Recurrent Networks
• Tailoring Artificial Neural Networks for Optimal Learning

Cited by in corpus (8)

• Learning strange attractors with reservoir systems
• Learn to Synchronize, Synchronize to Learn
• Quantum reservoir computing in finite dimensions
• Reservoir Computing for Macroeconomic Forecasting with Mixed Frequency Data
• Freezing chaos without synaptic plasticity
• Input-dependence in quantum reservoir computing
• Empirical Analysis of Limits for Memory Distance in Recurrent Neural Networks
• Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks