Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data
arXiv:2012.04859
Abstract
Kernels derived from deep neural networks (DNNs) in the infinite-width regime provide not only high performance in a range of machine learning tasks but also new theoretical insights into DNN training dynamics and generalization. In this paper, we extend the family of kernels associated with recurrent neural networks (RNNs), which were previously derived only for simple RNNs, to more complex architectures including bidirectional RNNs and RNNs with average pooling. We also develop a fast GPU implementation to exploit the full practical potential of the kernels. Though RNNs are typically only applied to time-series data, we demonstrate that classifiers using RNN-based kernels outperform a range of baseline methods on 90 non-time-series datasets from the UCI data repository.
References in corpus (17)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Deep Neural Networks as Gaussian Processes
- On Exact Computation with an Infinitely Wide Neural Net
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels
- Avoiding pathologies in very deep networks
- Enhanced Convolutional Neural Tangent Kernels
- Tensor Programs II: Neural Tangent Kernel for Any Architecture
- Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
- Infinite attention: NNGP and NTK for deep attention networks
- Recognition of basic hand movements using Electromyography
- Tensor Programs IIb: Architectural Universality of Neural Tangent Kernel Training Dynamics
- Chickenpox Cases in Hungary: a Benchmark Dataset for Spatiotemporal Signal Processing with Graph Neural Networks
- SymJAX: symbolic CPU/GPU/TPU programming
- Kernel-Based Smoothness Analysis of Residual Networks
- Learning and Generalization in RNNs