Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)

Abstract

Gradient descent yields zero training loss in polynomial time for deep neural networks despite non-convex nature of the objective function. The behavior of network in the infinite width limit trained by gradient descent can be described by the Neural Tangent Kernel (NTK) introduced in \cite{Jacot2018Neural}. In this paper, we study dynamics of the NTK for finite width Deep Residual Network (ResNet) using the neural tangent hierarchy (NTH) proposed in \cite{Huang2019Dynamics}. For a ResNet with smooth and Lipschitz activation function, we reduce the requirement on the layer width $m$ with respect to the number of training samples $n$ from quartic to cubic. Our analysis suggests strongly that the particular skip-connection structure of ResNet is the main reason for its triumph over fully-connected network.

arXiv comment72 pages, 1 figure

References in corpus (22)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Convergence Analysis of Two-layer Neural Networks with ReLU Activation
• How to Escape Saddle Points Efficiently
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Gradient Descent Finds Global Minima of Deep Neural Networks
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• The loss surface of deep and wide neural networks
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• On the Power of Over-parametrization in Neural Networks with Quadratic Activation
• Learning One-hidden-layer Neural Networks with Landscape Design
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Quadratic Suffices for Over-parametrization via Matrix Chernoff Bound
• A Priori Estimates of the Population Risk for Residual Networks
• Critical Points of Neural Networks: Analytical Forms and Landscape Properties
• Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima

Cited by in corpus (1)

• Phase Diagram of Initial Condensation for Two-layer Neural Networks