Determination of the edge of criticality in echo state networks through Fisher information maximization

Abstract

It is a widely accepted fact that the computational capability of recurrent neural networks is maximized on the so-called "edge of criticality". Once the network operates in this configuration, it performs efficiently on a specific application both in terms of (i) low prediction error and (ii) high short-term memory capacity. Since the behavior of recurrent networks is strongly influenced by the particular input signal driving the dynamics, a universal, application-independent method for determining the edge of criticality is still missing. In this paper, we aim at addressing this issue by proposing a theoretically motivated, unsupervised method based on Fisher information for determining the edge of criticality in recurrent neural networks. It is proven that Fisher information is maximized for (finite-size) systems operating in such critical regions. However, Fisher information is notoriously difficult to compute and either requires the probability density function or the conditional dependence of the system states with respect to the model parameters. The paper takes advantage of a recently-developed non-parametric estimator of the Fisher information matrix and provides a method to determine the critical region of echo state networks, a particular class of recurrent networks. The considered control parameters, which indirectly affect the echo state network performance, are explored to identify those configurations lying on the edge of criticality and, as such, maximizing Fisher information and computational performance. Experimental results on benchmarks and real-world data demonstrate the effectiveness of the proposed method.

References in corpus (8)

• On the difficulty of training Recurrent Neural Networks
• Data for Development: the D4D Challenge on Mobile Phone Data
• Transition to chaos in random networks with cell-type-specific connectivity
• Topological and Dynamical Complexity of Random Neural Networks
• Investigating echo state networks dynamics by means of recurrence analysis
• Dynamical criticality in the collective activity of a population of retinal neurons
• Empirical non-parametric estimation of the Fisher Information
• Cooperation, competition and the emergence of criticality in communities of adaptive systems

Cited by in corpus (20)

• Next Generation Reservoir Computing
• An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
• Optimal sequence memory in driven random networks
• Interpreting recurrent neural networks behaviour via excitable network attractors
• Exploiting Multiple Timescales in Hierarchical Echo State Networks
• Optimal short-term memory before the edge of chaos in driven random recurrent networks
• Integer Echo State Networks: Efficient Reservoir Computing for Digital Hardware
• Multiplex visibility graphs to investigate recurrent neural networks dynamics
• Memory and forecasting capacities of nonlinear recurrent networks
• Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere
• The Echo Index and multistability in input-driven recurrent neural networks
• Quantifying Relevance in Learning and Inference
• Local homeostatic regulation of the spectral radius of echo-state networks
• A characterization of the Edge of Criticality in Binary Echo State Networks
• Hyperparameter Tuning in Echo State Networks
• Fisher information flow in artificial neural networks
• Phase transitions from linear to nonlinear information processing in neural networks
• Criticality Analysis: Bio-inspired Nonlinear Data Representation
• Evolutionary aspects of Reservoir Computing
• Learnability Window in Gated Recurrent Neural Networks