Noisy Recurrent Neural Networks

Abstract

We provide a general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can be viewed as discretizations of stochastic differential equations driven by input data. This framework allows us to study the implicit regularization effect of general noise injection schemes by deriving an approximate explicit regularizer in the small noise regime. We find that, under reasonable assumptions, this implicit regularization promotes flatter minima; it biases towards models with more stable dynamics; and, in classification tasks, it favors models with larger classification margin. Sufficient conditions for global stability are obtained, highlighting the phenomenon of stochastic stabilization, where noise injection can improve stability during training. Our theory is supported by empirical results which demonstrate that the RNNs have improved robustness with respect to various input perturbations.

38 pages

References in corpus (16)

• On the difficulty of training Recurrent Neural Networks
• Recurrent Neural Network Regularization
• Understanding deep learning requires rethinking generalization
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
• Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning
• Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction
• Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Lipschitz Recurrent Neural Networks
• Stochastic Normalizing Flows
• MaxUp: A Simple Way to Improve Generalization of Neural Network Training
• Continuous-in-Depth Neural Networks
• Towards Robust ResNet: A Small Step but A Giant Leap
• Deep Adversarial Koopman Model for Reaction-Diffusion systems

Cited by in corpus (4)

• A Differential Geometry Perspective on Orthogonal Recurrent Models
• Long Expressive Memory for Sequence Modeling
• Cluster-and-Conquer: A Framework For Time-Series Forecasting
• Noisy Feature Mixup