A Precise Performance Analysis of Learning with Random Features

Abstract

We study the problem of learning an unknown function using random feature models. Our main contribution is an exact asymptotic analysis of such learning problems with Gaussian data. Under mild regularity conditions for the feature matrix, we provide an exact characterization of the asymptotic training and generalization errors, valid in both the under-parameterized and over-parameterized regimes. The analysis presented in this paper holds for general families of feature matrices, activation functions, and convex loss functions. Numerical results validate our theoretical predictions, showing that our asymptotic findings are in excellent agreement with the actual performance of the considered learning problem, even in moderate dimensions. Moreover, they reveal an important role played by the regularization, the loss function and the activation function in the mitigation of the "double descent phenomenon" in learning.

References in corpus (6)

• Reconciling modern machine learning practice and the bias-variance trade-off
• The generalization error of random features regression: Precise asymptotics and double descent curve
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Generalisation error in learning with random features and the hidden manifold model
• Optimal Regularization Can Mitigate Double Descent
• The Spectrum of Random Inner-product Kernel Matrices

Cited by in corpus (15)

• Learning curves of generic features maps for realistic datasets with a teacher-student model
• The Gaussian equivalence of generative models for learning with shallow neural networks
• Universality Laws for High-Dimensional Learning with Random Features
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• On the Inherent Regularization Effects of Noise Injection During Training
• Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
• Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
• Label-Imbalanced and Group-Sensitive Classification under Overparameterization
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Strong replica symmetry for high-dimensional disordered log-concave Gibbs measures
• Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models
• On the Double Descent of Random Features Models Trained with SGD
• Appearance of Random Matrix Theory in Deep Learning
• Sharp global convergence guarantees for iterative nonconvex optimization: A Gaussian process perspective
• Harmless interpolation in regression and classification with structured features