Implicit Bias of Linear RNNs
arXiv:2101.07833
Abstract
Contemporary wisdom based on empirical studies suggests that standard recurrent neural networks (RNNs) do not perform well on tasks requiring long-term memory. However, precise reasoning for this behavior is still unknown. This paper provides a rigorous explanation of this property in the special case of linear RNNs. Although this work is limited to linear RNNs, even these systems have traditionally been difficult to analyze due to their non-linear parameterization. Using recently-developed kernel regime analysis, our main result shows that linear RNNs learned from random initializations are functionally equivalent to a certain weighted 1D-convolutional network. Importantly, the weightings in the equivalent model cause an implicit bias to elements with smaller time lags in the convolution and hence, shorter memory. The degree of this bias depends on the variance of the transition kernel matrix at initialization and is related to the classic exploding and vanishing gradients problem. The theory is validated in both synthetic and real data experiments.
arXiv comment30 pages, 4 figures
References in corpus (20)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Full-Capacity Unitary Recurrent Neural Networks
- A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
- Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
- The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
- SGD Learns the Conjugate Kernel Class of the Network
- RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference
- Asymptotic errors for convex penalized linear regression beyond Gaussian matrices
- The Recurrent Neural Tangent Kernel
- Input-Output Equivalence of Unitary and Contractive RNNs