The Slow Deterioration of the Generalization Error of the Random Feature Model
arXiv:2008.05621
Abstract
The random feature model exhibits a kind of resonance behavior when the number of parameters is close to the training sample size. This behavior is characterized by the appearance of large generalization gap, and is due to the occurrence of very small eigenvalues for the associated Gram matrix. In this paper, we examine the dynamic behavior of the gradient descent algorithm in this regime. We show, both theoretically and experimentally, that there is a dynamic self-correction mechanism at work: The larger the eventual generalization gap, the slower it develops, both because of the small eigenvalues. This gives us ample time to stop the training process and obtain solutions with good generalization property.
References in corpus (8)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- The generalization error of random features regression: Precise asymptotics and double descent curve
- A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
- Diverse Neural Network Learns True Target Functions
- Understanding training and generalization in deep learning by Fourier analysis
- Learning with SGD and Random Features
- Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks