Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

Abstract

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

References in corpus (5)

• DropBlock: A regularization method for convolutional networks
• Shake-Shake regularization
• Analyzing Inverse Problems with Invertible Neural Networks
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint

Cited by in corpus (32)

• Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective
• KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics
• Scalable Gradients for Stochastic Differential Equations
• Bayesian Learning-Based Adaptive Control for Safety Critical Systems
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• Stochastic Normalizing Flows
• TrajectoryNet: A Dynamic Optimal Transport Network for Modeling Cellular Dynamics
• On Robustness of Neural Ordinary Differential Equations
• Continuous-in-Depth Neural Networks
• Stochastic Normalizing Flows
• STEER: Simple Temporal Regularization For Neural ODEs
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Beltrami Flow and Neural Diffusion on Graphs
• Robust Asymmetric Heterogeneous Federated Learning with Corrupted Clients
• A Cost-Efficient FPGA Implementation of Tiny Transformer Model using Neural ODE
• Neural SDEs as Infinite-Dimensional GANs
• Stochasticity in Neural ODEs: An Empirical Study
• Differential Bayesian Neural Nets
• Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics
• Policy Analysis using Synthetic Controls in Continuous-Time
• Generative modeling of time-dependent densities via optimal transport and projection pursuit
• Interpolation between Residual and Non-Residual Networks
• Meta-Solver for Neural Ordinary Differential Equations
• Efficient and Accurate Gradients for Neural SDEs
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks
• Deep limits and cut-off phenomena for neural networks
• Deep Latent-Variable Kernel Learning
• Federated Learning as a Mean-Field Game
• Time-changed normalizing flows for accurate SDE modeling
• Robust and Scalable SDE Learning: A Functional Perspective
• Neural Delay Differential Equations
• E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations