Benign Overfitting and Noisy Features

Abstract

Modern machine learning often operates in the regime where the number of parameters is much higher than the number of data points, with zero training loss and yet good generalization, thereby contradicting the classical bias-variance trade-off. This \textit{benign overfitting} phenomenon has recently been characterized using so called \textit{double descent} curves where the risk undergoes another descent (in addition to the classical U-shaped learning curve when the number of parameters is small) as we increase the number of parameters beyond a certain threshold. In this paper, we examine the conditions under which \textit{Benign Overfitting} occurs in the random feature (RF) models, i.e. in a two-layer neural network with fixed first layer weights. We adopt a new view of random feature and show that \textit{benign overfitting} arises due to the noise which resides in such features (the noise may already be present in the data and propagate to the features or it may be added by the user to the features directly) and plays an important implicit regularization role in the phenomenon.

References in corpus (15)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Benign Overfitting in Linear Regression
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences
• Generalization Properties of Learning with Random Features
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Deep Convolutional Networks as shallow Gaussian Processes
• Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guarantees
• SGD Learns the Conjugate Kernel Class of the Network
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent

Cited by in corpus (5)

• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Failures of model-dependent generalization bounds for least-norm interpolation
• Towards an Understanding of Benign Overfitting in Neural Networks
• The Geometry of Over-parameterized Regression and Adversarial Perturbations