The Recurrent Neural Tangent Kernel

Abstract

The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.

References in corpus (18)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
• On the Convergence Rate of Training Recurrent Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Avoiding pathologies in very deep networks
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
• Tensor Programs III: Neural Matrix Laws

Cited by in corpus (4)

• Tensor Programs III: Neural Matrix Laws
• Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory
• Experiments with Rich Regime Training for Deep Learning
• Implicit Bias of Linear RNNs