The difference between memory and prediction in linear recurrent networks

Abstract

Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node networks optimized for prediction are nearly at upper bounds on predictive capacity given by Wiener filters, and are roughly equivalent in performance to randomly generated five-node networks. Our results suggest that maximizing memory capacity leads to very different networks than maximizing predictive capacity, and that optimizing recurrent weights can decrease reservoir size by half an order of magnitude.

References in corpus (3)

• Short-Term Memory in Orthogonal Neural Networks
• Nearly extensive sequential memory lifetime achieved by coupled nonlinear neurons
• Memory and Information Processing in Recurrent Neural Networks

Cited by in corpus (15)

• Using a reservoir computer to learn chaotic attractors, with applications to chaos synchronisation and cryptography
• Using reservoir computers to distinguish chaotic signals
• Network Structure Effects in Reservoir Computers
• Coupled nonlinear delay systems as deep convolutional neural networks
• Scalable photonic platform for real-time quantum reservoir computing
• Exploiting Multiple Timescales in Hierarchical Echo State Networks
• Optimal short-term memory before the edge of chaos in driven random recurrent networks
• Learn to Synchronize, Synchronize to Learn
• Memory and forecasting capacities of nonlinear recurrent networks
• Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere
• Risk bounds for reservoir computing
• Stability Analysis of Reservoir Computers Dynamics via Lyapunov Functions
• Time cells might be optimized for predictive capacity, not redundancy reduction or memory capacity
• New analytic formulae for memory and prediction functions in reservoir computers with time delays
• Memory Uncertainty Relation and Harmonic Memory in Random Recurrent Networks