Framing RNN as a kernel method: A neural ODE approach
arXiv:2106.01202
Abstract
Building on the interpretation of a recurrent neural network (RNN) as a continuous-time neural differential equation, we show, under appropriate conditions, that the solution of a RNN can be viewed as a linear function of a specific feature set of the input sequence, known as the signature. This connection allows us to frame a RNN as a kernel method in a suitable reproducing kernel Hilbert space. As a consequence, we obtain theoretical guarantees on generalization and stability for a large class of recurrent networks. Our results are illustrated on simulated datasets.
arXiv comment33 pages, 7 figures, accepted for an oral presentation at NeurIPS 2021
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Natural Language Processing (almost) from Scratch
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series
- Neural Controlled Differential Equations for Irregular Time Series
- AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
- Rough paths, Signatures and the modelling of functions on streams
- Learning from the past, predicting the statistics for the future, learning an evolving system
- Learning Differential Equations that are Easy to Solve
- Signatory: differentiable computations of the signature and logsignature transforms, on both CPU and GPU
- Deep Signature Transforms
- Lipschitz Recurrent Neural Networks
- POPQORN: Quantifying Robustness of Recurrent Neural Networks
- A Kernel Perspective for Regularizing Deep Neural Networks
- Learning stochastic differential equations using RNN with log signature features
- Neural Rough Differential Equations for Long Time Series
- Bayesian Learning from Sequential Data using Gaussian Processes with Signature Covariances
- Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition
- Derivatives pricing using signature payoffs
- Sample Complexity Bounds for Recurrent Neural Networks with Application to Combinatorial Graph Problems