Noisy Recurrent Neural Networks
arXiv:2102.04877
Abstract
We provide a general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can be viewed as discretizations of stochastic differential equations driven by input data. This framework allows us to study the implicit regularization effect of general noise injection schemes by deriving an approximate explicit regularizer in the small noise regime. We find that, under reasonable assumptions, this implicit regularization promotes flatter minima; it biases towards models with more stable dynamics; and, in classification tasks, it favors models with larger classification margin. Sufficient conditions for global stability are obtained, highlighting the phenomenon of stochastic stabilization, where noise injection can improve stability during training. Our theory is supported by empirical results which demonstrate that the RNNs have improved robustness with respect to various input perturbations.
arXiv comment38 pages
References in corpus (33)
- On the difficulty of training Recurrent Neural Networks
- Recurrent Neural Network Regularization
- Understanding deep learning requires rethinking generalization
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
- Hamiltonian Neural Networks
- Strong convergence of an explicit numerical method for SDEs with nonglobally Lipschitz continuous coefficients
- Robust Large Margin Deep Neural Networks
- Numerical approximations of stochastic differential equations with non-globally Lipschitz continuous coefficients
- Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
- Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
- Stable Recurrent Models
- AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
- Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
- Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning
- Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks
- Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction
- Improving performance of recurrent neural network with relu nonlinearity
- Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
- Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don't
- Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
- Learning Compositional Koopman Operators for Model-Based Control
- Theory of Deep Learning III: explaining the non-overfitting puzzle
- Lipschitz Recurrent Neural Networks
- Stochastic Normalizing Flows
- The Implicit Regularization of Stochastic Gradient Flow for Least Squares
- MaxUp: A Simple Way to Improve Generalization of Neural Network Training
- Continuous-in-Depth Neural Networks
- Noisin: Unbiased Regularization for Recurrent Neural Networks
- Stochastic Training of Residual Networks: a Differential Equation Viewpoint
- Towards Robust ResNet: A Small Step but A Giant Leap
- An introduction to SDE simulation
- Deep Adversarial Koopman Model for Reaction-Diffusion systems