Learning curves for Gaussian process regression with power-law priors and targets
arXiv:2110.12231
Abstract
We characterize the power-law asymptotics of learning curves for Gaussian process regression (GPR) under the assumption that the eigenspectrum of the prior and the eigenexpansion coefficients of the target function follow a power law. Under similar assumptions, we leverage the equivalence between GPR and kernel ridge regression (KRR) to show the generalization error of KRR. Infinitely wide neural networks can be related to GPR with respect to the neural network GP kernel and the neural tangent kernel, which in several cases is known to have a power-law spectrum. Hence our methods can be applied to study the generalization error of infinitely wide neural networks. We present toy experiments demonstrating the theory.
arXiv comment76 pages, 7 table, 6 figure
References in corpus (15)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Deep Neural Networks as Gaussian Processes
- Deep Learning Scaling is Predictable, Empirically
- On Exact Computation with an Infinitely Wide Neural Net
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Explaining Neural Scaling Laws
- The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
- Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
- A Fine-Grained Spectral Perspective on Neural Networks
- Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
- Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration
- Relative concentration bounds for the spectrum of kernel matrices
- Universal scaling laws in the gradient descent training of neural networks