Random features and polynomial rules

Abstract

Random features models play a distinguished role in the theory of deep learning, describing the behavior of neural networks close to their infinite-width limit. In this work, we present a thorough analysis of the generalization performance of random features models for generic supervised learning problems with Gaussian data. Our approach, built with tools from the statistical mechanics of disordered systems, maps the random features model to an equivalent polynomial model, and allows us to plot average generalization curves as functions of the two main control parameters of the problem: the number of random features $N$ and the size $P$ of the training set, both assumed to scale as powers in the input dimension $D$. Our results extend the case of proportional scaling between $N$, $P$ and $D$. They are in accordance with rigorous bounds known for certain particular learning tasks and are in quantitative agreement with numerical experiments performed over many order of magnitudes of $N$ and $P$. We find good agreement also far from the asymptotic limits where $D\to \infty$ and at least one between $P/D^K$, $N/D^L$ remains finite.

11 pages + appendix, 4 figures

References in corpus (27)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Introduction to Random Matrices - Theory and Practice
• Classification and Geometry of General Perceptual Manifolds
• Statistical Mechanics of Support Vector Networks
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• The spectrum of kernel random matrices
• Generalisation error in learning with random features and the hidden manifold model
• Rethinking mean-field glassy dynamics and its relation with the energy landscape: the awkward case of the spherical mixed p-spin model
• Linear Readout of Object Manifolds
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Critical jammed phase of the linear perceptron
• Statistical learning theory of structured data
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
• Counting the learnable functions of structured data
• Beyond the storage capacity: data driven satisfiability transition
• Generalization from correlated sets of patterns in the perceptron
• Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression
• Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
• Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime
• Restoring balance: principled under/oversampling of data for optimal classification
• Exact results on high-dimensional linear regression via statistical physics
• Asymptotics of Learning with Deep Structured (Random) Features
• An Equivalence Principle for the Spectrum of Random Inner-Product Kernel Matrices with Polynomial Scalings
• Dimension-free deterministic equivalents and scaling laws for random feature regression
• Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime
• Risk and cross validation in ridge regression with correlated samples

Cited by in corpus (3)

• Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation
• Statistical mechanics of extensive-width Bayesian neural networks near interpolation
• Optimal generalisation and learning transition in extensive-width shallow neural networks near interpolation