Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective

Abstract

Deep residual networks (ResNets) have demonstrated better generalization performance than deep feedforward networks (FFNets). However, the theory behind such a phenomenon is still largely unknown. This paper studies this fundamental problem in deep learning from a so-called "neural tangent kernel" perspective. Specifically, we first show that under proper conditions, as the width goes to infinity, training deep ResNets can be viewed as learning reproducing kernel functions with some kernel function. We then compare the kernel of deep ResNets with that of deep FFNets and discover that the class of functions induced by the kernel of FFNets is asymptotically not learnable, as the depth goes to infinity. In contrast, the class of functions induced by the kernel of ResNets does not exhibit such degeneracy. Our discovery partially justifies the advantages of deep ResNets over deep FFNets in generalization abilities. Numerical results are provided to support our claim.

arXiv commentAccepted in NeurIPS 2020

References in corpus (18)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Training Very Deep Networks
• Recent Trends in Deep Learning Based Natural Language Processing
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Convolutional Kernel Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Deep Convolutional Networks as shallow Gaussian Processes
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Towards Understanding the Importance of Shortcut Connections in Residual Networks
• Can SGD Learn Recurrent Neural Networks with Provable Generalization?
• Algorithm-Dependent Generalization Bounds for Overparameterized Deep Residual Networks

Cited by in corpus (9)

• A Theoretical Analysis of Deep Q-Learning
• Wide and Deep Neural Networks Achieve Optimality for Classification
• On the Similarity between the Laplace and Neural Tangent Kernels
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations
• The Recurrent Neural Tangent Kernel
• Kernel-Based Smoothness Analysis of Residual Networks
• A Fully Spiking Hybrid Neural Network for Energy-Efficient Object Detection
• Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks