Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory

Abstract

Recurrent neural networks (RNNs) are brain-inspired models widely used in machine learning for analyzing sequential data. The present work is a contribution towards a deeper understanding of how RNNs process input signals using the response theory from nonequilibrium statistical mechanics. For a class of continuous-time stochastic RNNs (SRNNs) driven by an input signal, we derive a Volterra type series representation for their output. This representation is interpretable and disentangles the input signal from the SRNN architecture. The kernels of the series are certain recursively defined correlation functions with respect to the unperturbed dynamics that completely determine the output. Exploiting connections of this representation and its implications to rough paths theory, we identify a universal feature -- the response feature, which turns out to be the signature of tensor product of the input signal and a natural support basis. In particular, we show that SRNNs, with only the weights in the readout layer optimized and the weights in the hidden layer kept fixed and not optimized, can be viewed as kernel machines operating on a reproducing kernel Hilbert space associated with the response feature.

arXiv comment48 pages

References in corpus (21)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• On the difficulty of training Recurrent Neural Networks
• Recent Advances in Physical Reservoir Computing: A Review
• How to Construct Deep Recurrent Neural Networks
• GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
• An update on nonequilibrium linear response
• The Random Feature Model for Input-Output Maps between Banach Spaces
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Symplectic Recurrent Neural Networks
• Rough paths, Signatures and the modelling of functions on streams
• Learning from the past, predicting the statistics for the future, learning an evolving system
• Lipschitz Recurrent Neural Networks
• Learning stochastic differential equations using RNN with log signature features
• Neural Rough Differential Equations for Long Time Series
• Recurrent Neural Networks in the Eye of Differential Equations
• Linear response in neuronal networks: from neurons dynamics to collective response
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint
• Bayesian Learning from Sequential Data using Gaussian Processes with Signature Covariances
• Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies
• Mathematical foundation of nonequilibrium fluctuation-dissipation theorems for inhomogeneous diffusion processes with unbounded coefficients
• The Recurrent Neural Tangent Kernel

Cited by in corpus (4)

• Predicting Critical Transitions in Multiscale Dynamical Systems Using Reservoir Computing
• Continuous-in-Depth Neural Networks
• Response Theory Identifies Reaction Coordinates and Explains Critical Phenomena in Noisy Interacting Systems
• On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis