Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks

Abstract

Recent works have demonstrated that increasing model capacity through width in over-parameterized neural networks leads to a decrease in test risk. For neural networks, however, model capacity can also be increased through depth, yet understanding the impact of increasing depth on test risk remains an open question. In this work, we demonstrate that the test risk of over-parameterized convolutional networks is a U-shaped curve (i.e. monotonically decreasing, then increasing) with increasing depth. We first provide empirical evidence for this phenomenon via image classification experiments using both ResNets and the convolutional neural tangent kernel (CNTK). We then present a novel linear regression framework for characterizing the impact of depth on test risk, and show that increasing depth leads to a U-shaped test risk for the linear CNTK. In particular, we prove that the linear CNTK corresponds to a depth-dependent linear transformation on the original space and characterize properties of this transformation. We then analyze over-parameterized linear regression under arbitrary linear transformations and, in simplified settings, provably identify the depths which minimize each of the bias and variance terms of the test risk.

arXiv comment27 pages, 23 figures

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Benign Overfitting in Linear Regression
• On Exact Computation with an Infinitely Wide Neural Net
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• Optimization Landscape and Expressivity of Deep CNNs
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
• Memorization in Overparameterized Autoencoders
• Disentangling Trainability and Generalization in Deep Neural Networks
• The Low-Rank Simplicity Bias in Deep Networks
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm

Cited by in corpus (2)

• Wide and Deep Neural Networks Achieve Optimality for Classification
• The Low-Rank Simplicity Bias in Deep Networks