On Generalization Bounds of a Family of Recurrent Neural Networks

Abstract

Recurrent Neural Networks (RNNs) have been widely applied to sequential data analysis. Due to their complicated modeling structures, however, the theory behind is still largely missing. To connect theory and practice, we study the generalization properties of vanilla RNNs as well as their variants, including Minimal Gated Unit (MGU), Long Short Term Memory (LSTM), and Convolutional (Conv) RNNs. Specifically, our theory is established under the PAC-Learning framework. The generalization bound is presented in terms of the spectral norms of the weight matrices and the total number of parameters. We also establish refined generalization bounds with additional norm assumptions, and draw a comparison among these bounds. We remark: (1) Our generalization bound for vanilla RNNs is significantly tighter than the best of existing results; (2) We are not aware of any other generalization bounds for MGU, LSTM, and Conv RNNs in the exiting literature; (3) We demonstrate the advantages of these variants in generalization.

arXiv comment30 pages, 5 figures

References in corpus (13)

• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Sequence Transduction with Recurrent Neural Networks
• Understanding deep learning requires rethinking generalization
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• DRAW: A Recurrent Neural Network For Image Generation
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Learning to Diagnose with LSTM Recurrent Neural Networks
• Pointer Sentinel Mixture Models
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• On orthogonality and learning recurrent networks with long term dependencies
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization

Cited by in corpus (9)

• Machine Learning for Prediction with Missing Dynamics
• Generalization and Representational Limits of Graph Neural Networks
• Recent advances in deep learning theory
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• Can SGD Learn Recurrent Neural Networks with Provable Generalization?
• PAC-Bayes Generalisation Bounds for Dynamical Systems Including Stable RNNs
• Norm-based generalisation bounds for multi-class convolutional neural networks
• Understanding Deep Architectures with Reasoning Layer
• Spectral Pruning for Recurrent Neural Networks