Towards Understanding the Importance of Shortcut Connections in Residual Networks
arXiv:1909.04653
Abstract
Residual Network (ResNet) is undoubtedly a milestone in deep learning. ResNet is equipped with shortcut connections between layers, and exhibits efficient training using simple first order algorithms. Despite of the great empirical success, the reason behind is far from being well understood. In this paper, we study a two-layer non-overlapping convolutional ResNet. Training such a network requires solving a non-convex optimization problem with a spurious local optimum. We show, however, that gradient descent combined with proper normalization, avoids being trapped by the spurious local optimum, and converges to a global optimum in polynomial time, when the weight of the first layer is initialized at 0, and that of the second layer is initialized arbitrarily in a ball. Numerical experiments are provided to support our theory.
arXiv commentThirty-third Conference on Neural Information Processing Systems, 2019
Cited by in corpus (8)
- Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective
- Continuous-in-Depth Neural Networks
- A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth
- Learning Frequency Domain Approximation for Binary Neural Networks
- On a Sparse Shortcut Topology of Artificial Neural Networks
- On the Demystification of Knowledge Distillation: A Residual Network Perspective
- ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees
- Augmented Shortcuts for Vision Transformers