Learning curves of generic features maps for realistic datasets with a teacher-student model

Abstract

Teacher-student models provide a framework in which the typical-case performance of high-dimensional supervised learning can be described in closed form. The assumptions of Gaussian i.i.d. input data underlying the canonical teacher-student model may, however, be perceived as too restrictive to capture the behaviour of realistic data sets. In this paper, we introduce a Gaussian covariate generalisation of the model where the teacher and student can act on different spaces, generated with fixed, but generic feature maps. While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework. Our contribution is then two-fold: First, we prove a rigorous formula for the asymptotic training loss and generalisation error. Second, we present a number of situations where the learning curve of the model captures the one of a realistic data set learned with kernel regression and classification, with out-of-the-box feature maps such as random projections or scattering transforms, or with pre-learned ones - such as the features learned by training multi-layer neural networks. We discuss both the power and the limitations of the framework.

arXiv commentv3: NeurIPS camera-ready

References in corpus (35)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Generative Adversarial Networks
• Message Passing Algorithms for Compressed Sensing
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Benign Overfitting in Linear Regression
• Statistical physics of inference: Thresholds and algorithms
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Deterministic equivalents for certain functionals of large random matrices
• Two models of double descent for weak features
• Statistical Mechanics of Support Vector Networks
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• The spectrum of kernel random matrices
• Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond
• A framework to characterize performance of LASSO algorithms
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Generalisation error in learning with random features and the hidden manifold model
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• When Do Neural Networks Outperform Kernel Methods?
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• The distribution of the Lasso: Uniform control over sparse balls and adaptive parameter tuning
• The Gaussian equivalence of generative models for learning with shallow neural networks
• The Lasso with general Gaussian designs with applications to hypothesis testing
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Kernel Alignment Risk Estimator: Risk Prediction from Training Data
• Universality Laws for High-Dimensional Learning with Random Features
• A Precise Performance Analysis of Learning with Random Features
• Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixtures
• On the convergence of the extremal eigenvalues of empirical covariance matrices with dependence
• Concentration of Measure and Large Random Matrices with an application to Sample Covariance Matrices
• Kernel regression in high dimensions: Refined analysis beyond double descent
• Large scale analysis of generalization error in learning using margin based classification methods

Cited by in corpus (18)

• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Mapping of attention mechanisms to a generalized Potts model
• Data-driven emergence of convolutional structure in neural networks
• Universality Laws for High-Dimensional Learning with Random Features
• Gaussian Universality of Perceptrons with Random Labels
• Learning curves for the multi-class teacher-student perceptron
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
• Asymptotic theory of in-context learning by linear attention
• Theoretical characterization of uncertainty in high-dimensional linear classification
• Penalization-induced shrinking without rotation in high dimensional GLM regression: a cavity analysis
• Neural-prior stochastic block model
• Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
• Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
• Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Out-of-Distribution Generalization in Kernel Regression
• Statistical physics analysis of graph neural networks: Approaching optimality in the contextual stochastic block model