Robust learning with implicit residual networks

Abstract

In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to the improved stability of both forward and backward propagations, has a favorable impact on the generalization power and allows to control the robustness of the network with only a few hyperparameters. In addition, the proposed reformulation of ResNet does not introduce new parameters and can potentially lead to a reduction in the number of required layers due to improved forward stability. Finally, we derive the memory-efficient training algorithm, propose a stochastic regularization technique and provide numerical results in support of our findings.

References in corpus (19)

• Spectral Normalization for Generative Adversarial Networks
• Improved Training of Wasserstein GANs
• Resnet in Resnet: Generalizing Residual Architectures
• Neural Ordinary Differential Equations
• Stable Architectures for Deep Neural Networks
• Robust Large Margin Deep Neural Networks
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activations
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks
• The Expressive Power of Neural Networks: A View from the Width
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• How to train your neural ODE: the world of Jacobian and kinetic regularization
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Learning Differential Equations that are Easy to Solve
• Augmented Neural ODEs
• Robust Learning with Jacobian Regularization
• IMEXnet: A Forward Stable Deep Neural Network
• Robust learning with implicit residual networks

Cited by in corpus (7)

• Time-series learning of latent-space dynamics for reduced-order model closure
• Robust learning with implicit residual networks
• Implicit Normalizing Flows
• A Shooting Formulation of Deep Learning
• Interpolation between Residual and Non-Residual Networks
• Stability of implicit neural networks for long-term forecasting in dynamical systems
• Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations