On Robustness of Neural Ordinary Differential Equations

Abstract

Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks.

References in corpus (6)

• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise
• Augmented Neural ODEs
• Adversarial Defense via Data Dependent Activation Function and Total Variation Minimization
• ResNets Ensemble via the Feynman-Kac Formalism to Improve Natural and Robust Accuracies

Cited by in corpus (18)

• Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective
• How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses
• Dissecting Neural ODEs
• Learning Differential Equations that are Easy to Solve
• Continuous-in-Depth Neural Networks
• STEER: Simple Temporal Regularization For Neural ODEs
• Improving Robustness and Uncertainty Modelling in Neural Ordinary Differential Equations
• MALI: A memory efficient and reverse accurate integrator for Neural ODEs
• CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection
• Learning Stable Galerkin Models of Turbulence with Differentiable Programming
• ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations
• LT-OCF: Learnable-Time ODE-based Collaborative Filtering
• Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows
• Adversarial Robustness of Stabilized NeuralODEs Might be from Obfuscated Gradients
• Interpolation between Residual and Non-Residual Networks
• Meta-Solver for Neural Ordinary Differential Equations
• Worrisome Properties of Neural Network Controllers and Their Symbolic Representations
• STR-GODEs: Spatial-Temporal-Ridership Graph ODEs for Metro Ridership Prediction