Investigating echo state networks dynamics by means of recurrence analysis

Abstract

In this paper, we elaborate over the well-known interpretability issue in echo state networks. The idea is to investigate the dynamics of reservoir neurons with time-series analysis techniques taken from research on complex systems. Notably, we analyze time-series of neuron activations with Recurrence Plots (RPs) and Recurrence Quantification Analysis (RQA), which permit to visualize and characterize high-dimensional dynamical systems. We show that this approach is useful in a number of ways. First, the two-dimensional representation offered by RPs provides a way for visualizing the high-dimensional dynamics of a reservoir. Our results suggest that, if the network is stable, reservoir and input denote similar line patterns in the respective RPs. Conversely, the more unstable the ESN, the more the RP of the reservoir presents instability patterns. As a second result, we show that the ${\mathrm{L}}_{\max }$ measure is highly correlated with the well-established maximal local Lyapunov exponent. This suggests that complexity measures based on RP diagonal lines distribution provide a valuable tool to quantify the degree of network stability. Finally, our analysis shows that all RQA measures fluctuate on the proximity of the so-called edge of stability, where an ESN typically achieves maximum computational capability. We verify that the determination of the edge of stability provided by such RQA measures is more accurate than two well-known criteria based on the Jacobian matrix of the reservoir. Therefore, we claim that RPs and RQA-based analyses can be used as valuable tools to design an effective network given a specific problem.

Revised version. 24 pages; 12 figures

References in corpus (3)

• Recurrence Plots for the Analysis of Complex Systems
• Nonlinear time-series analysis revisited
• Line Structures in Recurrence Plots

Cited by in corpus (15)

• An overview and comparative analysis of Recurrent Neural Networks for Short Term Load Forecasting
• Determination of the edge of criticality in echo state networks through Fisher information maximization
• Interpreting recurrent neural networks behaviour via excitable network attractors
• Integer Echo State Networks: Efficient Reservoir Computing for Digital Hardware
• Multiplex visibility graphs to investigate recurrent neural networks dynamics
• Learn to Synchronize, Synchronize to Learn
• Genesis of Basic and Multi-Layer Echo State Network Recurrent Autoencoders for Efficient Data Representations
• Learning representations for multivariate time series with missing data using Temporal Kernelized Autoencoders
• A journey in ESN and LSTM visualisations on a language task
• Temporal Convolution Derived Multi-Layered Reservoir Computing
• Potentials and Limitations of Deep Neural Networks for Cognitive Robots
• Excitatory/Inhibitory Balance Emerges as a Key Factor for RBN Performance, Overriding Attractor Dynamics
• Hierarchical Bi-level Multi-Objective Evolution of Single- and Multi-layer Echo State Network Autoencoders for Data Representations
• Randomly Weighted, Untrained Neural Tensor Networks Achieve Greater Relational Expressiveness
• The Expressivity and Training of Deep Neural Networks: toward the Edge of Chaos?