Stable Architectures for Deep Neural Networks

Abstract

Deep neural networks have become invaluable tools for supervised machine learning, e.g., classification of text or images. While often offering superior results over traditional techniques and successfully expressing complicated patterns in data, deep architectures are known to be challenging to design and train such that they generalize well to new data. Important issues with deep architectures are numerical instabilities in derivative-based learning algorithms commonly called exploding or vanishing gradients. In this paper we propose new forward propagation techniques inspired by systems of Ordinary Differential Equations (ODE) that overcome this challenge and lead to well-posed learning problems for arbitrarily deep networks. The backbone of our approach is our interpretation of deep learning as a parameter estimation problem of nonlinear dynamical systems. Given this formulation, we analyze stability and well-posedness of deep learning and use this new understanding to develop new network architectures. We relate the exploding and vanishing gradient phenomenon to the stability of the discrete ODE and present several strategies for stabilizing deep learning for very deep networks. While our new architectures restrict the solution space, several numerical experiments show their competitiveness with state-of-the-art networks.

23 pages, 7 figures

Cited by in corpus (57)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network
• Differentiable modeling to unify machine learning and physical models and advance Geosciences
• A deep learning enabler for non-intrusive reduced order modeling of fluid flows
• An artificial neural network framework for reduced order modeling of transient flows
• Deep Learning in Deterministic Computational Mechanics
• Deep Learning in Photoacoustic Tomography: Current approaches and future directions
• KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics
• Physics-Informed Probabilistic Learning of Linear Embeddings of Non-linear Dynamics With Guaranteed Stability
• How Deep Learning Sees the World: A Survey on Adversarial Attacks & Defenses
• The Random Feature Model for Input-Output Maps between Banach Spaces
• Time-series learning of latent-space dynamics for reduced-order model closure
• Continuous-time system identification with neural networks: Model structures and fitting criteria
• A Mean-Field Optimal Control Formulation of Deep Learning
• Multi-resolution partial differential equations preserved learning framework for spatiotemporal dynamics
• GFINNs: GENERIC Formalism Informed Neural Networks for Deterministic and Stochastic Dynamical Systems
• PDE-based Group Equivariant Convolutional Neural Networks
• Deep Neural Networks using a Single Neuron: Folded-in-Time Architecture using Feedback-Modulated Delay Loops
• Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problems
• Physics-Informed Supervised Residual Learning for Electromagnetic Modeling
• Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems
• Machine Learning from a Continuous Viewpoint
• Neural Born Iteration Method For Solving Inverse Scattering Problems: 2D Cases
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Neural Schrödinger Equation:Physical Law as Neural Network
• Accelerating Neural ODEs Using Model Order Reduction
• Physical deep learning based on optimal control of dynamical systems
• Sampling and resolution characteristics in reduced order models of shallow water equations: intrusive vs non-intrusive
• A literature survey of matrix methods for data science
• Neural integration for constitutive equations using small data
• An Efficient Parallel-in-Time Method for Optimization with Parabolic PDEs
• Operator Learning Using Random Features: A Tool for Scientific Computing
• Neural Fractional Differential Equations
• Robust learning with implicit residual networks
• CD-ROM: Complemented Deep-Reduced Order Model
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• State Space Representations of Deep Neural Networks
• Locally-symplectic neural networks for learning volume-preserving dynamics
• Personalized Algorithm Generation: A Case Study in Learning ODE Integrators
• Learning Complex Motion Plans using Neural ODEs with Safety and Stability Guarantees
• Optimal Dirichlet Boundary Control by Fourier Neural Operators Applied to Nonlinear Optics
• From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks
• Alternating the Population and Control Neural Networks to Solve High-Dimensional Stochastic Mean-Field Games
• GalaxyFlow: Upsampling Hydrodynamical Simulations for Realistic Mock Stellar Catalogs
• Emulating complex networks with a single delay differential equation
• Deep neural networks on diffeomorphism groups for optimal shape reparameterization
• Stability of Deep Neural Networks via discrete rough paths
• Quantized Convolutional Neural Networks Through the Lens of Partial Differential Equations
• A non-autonomous equation discovery method for time signal classification
• Numerical Stability of DeepGOPlus Inference
• Machine learning and serving of discrete field theories -- when artificial intelligence meets the discrete universe
• On the Geometric Mechanics of Assignment Flows for Metric Data Labeling
• On functions computed on trees
• Consistency and Variation in Kernel Neural Ranking Model
• Deep Neural Networks as Discrete Dynamical Systems: Implications for Physics-Informed Learning
• On $L^{\infty }$ stability for wave propagation and for linear inverse problems
• Using a Neural Network Classifier to Select Galaxies with the Most Accurate Photometric Redshifts