Learning Curve Theory

Abstract

Recently a number of empirical "universal" scaling law papers have been published, most notably by OpenAI. `Scaling laws' refers to power-law decreases of training or test error w.r.t. more data, larger neural networks, and/or more compute. In this work we focus on scaling w.r.t. data size $n$. Theoretical understanding of this phenomenon is largely lacking, except in finite-dimensional models for which error typically decreases with $n^{-1/2}$ or $n^{-1}$, where $n$ is the sample size. We develop and theoretically analyse the simplest possible (toy) model that can exhibit $n^{-\beta }$ learning curves for arbitrary power $\beta >0$, and determine whether power laws are universal or depend on the data distribution.

26 pages, 6 Figures

References in corpus (7)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Scaling Laws for Neural Language Models
• Deep Learning Scaling is Predictable, Empirically
• Scaling Laws for Autoregressive Generative Modeling
• A Bayesian View of the Poisson-Dirichlet Process
• Scaling Laws for Transfer
• A Neural Scaling Law from the Dimension of the Data Manifold

Cited by in corpus (6)

• Explaining Neural Scaling Laws
• Scaling Laws for Neural Machine Translation
• Scaling Scaling Laws with Board Games
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• Scaling Law for Recommendation Models: Towards General-purpose User Representations
• Turing-Universal Learners with Optimal Scaling Laws