Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network

Abstract

Overparametrized neural networks trained by gradient descent (GD) can provably overfit any training data. However, the generalization guarantee may not hold for noisy data. From a nonparametric perspective, this paper studies how well overparametrized neural networks can recover the true target function in the presence of random noises. We establish a lower bound on the $L_2$ estimation error with respect to the GD iterations, which is away from zero without a delicate scheme of early stopping. In turn, through a comprehensive analysis of ${\ell }_2$-regularized GD trajectories, we prove that for overparametrized one-hidden-layer ReLU neural network with the ${\ell }_2$ regularization: (1) the output is close to that of the kernel ridge regression with the corresponding neural tangent kernel; (2) minimax {optimal} rate of $L_2$ estimation error can be achieved. Numerical experiments confirm our theory and further demonstrate that the ${\ell }_2$ regularization approach improves the training robustness and works for a wider range of neural networks.

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• L2 Regularization versus Batch and Weight Normalization
• Early stopping and non-parametric regression: An optimal data-dependent stopping rule
• A Kernel Theory of Modern Data Augmentation
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• A characterization of Sobolev spaces on the sphere and an extension of Stolarsky's invariance principle to arbitrary smoothness
• On the training dynamics of deep networks with $L_2$ regularization
• Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
• Over-parametrized deep neural networks do not generalize well
• Optimal Nonparametric Inference via Deep Neural Network

Cited by in corpus (2)

• Nonparametric Regression with Shallow Overparameterized Neural Networks Trained by GD with Early Stopping
• Uniform Generalization Bounds for Overparameterized Neural Networks