Multi-level Residual Networks from Dynamical Systems View

Abstract

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet theoretically, such as unraveled view, unrolled iterative estimation and dynamical systems view. In this paper, we adopt the dynamical systems point of view, and analyze the lesioning properties of ResNet both theoretically and experimentally. Based on these analyses, we additionally propose a novel method for accelerating ResNet training. We apply the proposed method to train ResNets and Wide ResNets for three image classification benchmarks, reducing training time by more than 40% with superior or on-par accuracy.

Published as a conference paper at ICLR 2018

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• WaveNet: A Generative Model for Raw Audio
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Residual Connections Encourage Iterative Inference
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Sparsely Aggregated Convolutional Networks

Cited by in corpus (30)

• Data Driven Governing Equations Approximation Using Deep Neural Networks
• Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems
• Physics-guided deep learning framework for predictive modeling of the Reynolds stress anisotropy
• Scalable Gradients for Stochastic Differential Equations
• MgNet: A Unified Framework of Multigrid and Convolutional Neural Network
• Reversible Architectures for Arbitrarily Deep Residual Neural Networks
• Learning Differential Equations that are Easy to Solve
• NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations
• Hypersolvers: Toward Fast Continuous-Depth Models
• Stochastic Training of Residual Networks: a Differential Equation Viewpoint
• State Space Representations of Deep Neural Networks
• Hard Encoding of Physics for Learning Spatiotemporal Dynamics
• Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs
• Applications of Koopman Mode Analysis to Neural Networks
• ResNet After All? Neural ODEs and Their Numerical Solution
• Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning
• Functional Gradient Boosting based on Residual Network Perception
• Learning optimal multigrid smoothers via neural networks
• Meta-Solver for Neural Ordinary Differential Equations
• Non-intrusive inference reduced order model for fluids using linear multistep neural network
• Constrained Linear Data-feature Mapping for Image Classification
• Analytic Continuation of Noisy Data Using Adams Bashforth ResNet
• Multigrid-in-Channels Architectures for Wide Convolutional Neural Networks
• Closure Learning for Nonlinear Model Reduction Using Deep Residual Neural Network
• Machine Learning and Control Theory
• A Multilevel Approach to Training
• Training of deep residual networks with stochastic MG/OPT
• Solving Image PDEs with a Shallow Network
• Neural Delay Differential Equations
• Vector Field Neural Networks