NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations

Abstract

This paper introduces Non-Autonomous Input-Output Stable Network(NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and allows stability to be enforced so that blocks can be unrolled adaptively to a pattern-dependent processing depth. NAIS-Net induces non-trivial, Lipschitz input-output maps, even for an infinite unroll length. We prove that the network is globally asymptotically stable so that for every initial condition there is exactly one input-dependent equilibrium assuming $tanh$ units, and incrementally stable for ReL units. An efficient implementation that enforces the stability under derived conditions for both fully-connected and convolutional layers is also presented. Experimental results show how NAIS-Net exhibits stability in practice, yielding a significant reduction in generalization gap compared to ResNets.

NeurIPS 2018 (updated version): new results and proof on incremental stability for ReLU case

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Spectral Normalization for Generative Adversarial Networks
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Residual Connections Encourage Iterative Inference
• Exploring Neural Transducers for End-to-End Speech Recognition
• Multi-level Residual Networks from Dynamical Systems View
• A recurrent neural network without chaos
• Can recurrent neural networks warp time?

Cited by in corpus (14)

• Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network
• Invertible Residual Networks
• Neural Controlled Differential Equations for Irregular Time Series
• ANODE: Unconditionally Accurate Memory-Efficient Gradients for Neural ODEs
• Parameterized Neural Ordinary Differential Equations: Applications to Computational Physics Problems
• Lipschitz Recurrent Neural Networks
• ANODEV2: A Coupled Neural ODE Evolution Framework
• IMEXnet: A Forward Stable Deep Neural Network
• SNODE: Spectral Discretization of Neural ODEs for System Identification
• Safe Interactive Model-Based Learning
• Neural Lyapunov Model Predictive Control: Learning Safe Global Controllers from Sub-optimal Examples
• Residual networks classify inputs based on their neural transient dynamics
• Continuous-time Models for Stochastic Optimization Algorithms
• Robust Neural Networks inspired by Strong Stability Preserving Runge-Kutta methods