Analysis of the Gradient Descent Algorithm for a Deep Neural Network Model with Skip-connections

Abstract

The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path are also established. As a consequence, it is shown that when the target function is in the reproducing kernel Hilbert space (RKHS) with a kernel defined by the initialization, there exist generalizable early-stopping solutions along the GD path. In addition, it is also shown that the GD path is uniformly close to the functions given by the related random feature model. Consequently, in this "implicit regularization" setting, the deep neural network model deteriorates to a random feature model. Our results hold for neural networks of any width larger than the input dimension.

29 pages, 4 figures

References in corpus (14)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• SGD Learns the Conjugate Kernel Class of the Network
• Diverse Neural Network Learns True Target Functions
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• A Priori Estimates of the Population Risk for Residual Networks
• Learning with SGD and Random Features
• Width Provably Matters in Optimization for Deep Linear Neural Networks

Cited by in corpus (12)

• Optimization for deep learning: theory and algorithms
• A Priori Estimates of the Population Risk for Two-layer Neural Networks
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamics
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations
• Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernels
• Continuous vs. Discrete Optimization of Deep Neural Networks
• Global Convergence of Gradient Descent for Deep Linear Residual Networks
• Scaling Properties of Deep Residual Networks
• What training reveals about neural network complexity
• A Revision of Neural Tangent Kernel-based Approaches for Neural Networks