Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

Abstract

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

References in corpus (1)

• Shake-Shake regularization

Cited by in corpus (13)

• Stochastic Normalizing Flows
• Continuous-in-Depth Neural Networks
• Beltrami Flow and Neural Diffusion on Graphs
• Differential Bayesian Neural Nets
• Policy Analysis using Synthetic Controls in Continuous-Time
• Interpolation between Residual and Non-Residual Networks
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Meta-Solver for Neural Ordinary Differential Equations
• Federated Learning as a Mean-Field Game
• Deep Latent-Variable Kernel Learning
• Deep limits and cut-off phenomena for neural networks
• Robust and Scalable SDE Learning: A Functional Perspective
• Neural Delay Differential Equations