Explaining Neural Scaling Laws

Abstract

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origins of and relationships between scaling exponents.

11 pages, 3 figures + Supplement (expanded). This version to appear in PNAS

References in corpus (39)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Language Models are Few-Shot Learners
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Scaling Laws for Neural Language Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Deep Learning Scaling is Predictable, Empirically
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Scaling description of generalization with number of parameters in deep learning
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Scaling Laws for Autoregressive Generative Modeling
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Local polynomial regression on unknown manifolds
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Deep Convolutional Networks as shallow Gaussian Processes
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• The large learning rate phase of deep learning: the catapult mechanism
• A Constructive Prediction of the Generalization Error Across Scales
• Asymptotics of Wide Networks from Feynman Diagrams
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigm
• Neural Kernels Without Tangents
• The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalization
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition
• A Neural Scaling Law from the Dimension of the Data Manifold
• On the Predictability of Pruning Across Scales
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• Asymptotics of Wide Convolutional Neural Networks
• Distributional Generalization: A New Kind of Generalization
• Non-Gaussian processes and neural networks at finite widths
• A Solvable Model of Neural Scaling Laws
• More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize
• Learning Curve Theory
• Replica theory for learning curves for Gaussian processes on random graphs
• Implicit bias of deep linear networks in the large learning rate phase

Cited by in corpus (26)

• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Precision Machine Learning
• Scaling Laws for Deep Learning
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Scaling Laws for Neural Machine Translation
• Deep multi-task mining Calabi-Yau four-folds
• Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Multispectral airborne laser scanning for tree species classification: a benchmark of machine learning and deep learning algorithms
• Tensor network to learn the wavefunction of data
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• Universal scaling laws in the gradient descent training of neural networks
• A Hybrid Two-level MCMC Framework to Accelerate Posterior Mean Estimation with Deep Learning Surrogates for Bayesian Inverse Problems
• Bayesian RG Flow in Neural Network Field Theories
• Uncertainty Quantification From Scaling Laws in Deep Neural Networks
• Scaling Law for Recommendation Models: Towards General-purpose User Representations
• Wilsonian Renormalization of Neural Network Gaussian Processes
• Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis
• Neural Scaling Laws for Deep Regression
• Family-Vicsek universality of the binary intrinsic dimension of nonequilibrium data
• Learning curves for Gaussian process regression with power-law priors and targets
• Learning Curves for SGD on Structured Features
• Network Dynamics-Based Framework for Understanding Deep Neural Networks
• Harnessing intuitive local evolution rules for physical learning
• A solution to generalized learning from small training sets found in infants repeated visual experiences of individual objects
• Turing-Universal Learners with Optimal Scaling Laws