Scaling Laws for Autoregressive Generative Modeling

Abstract

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$)+D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8\times 8$ resolution, and we can forecast the model size needed to achieve any given reducible loss (ie $D_{\mathrm{KL}}$) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

20+17 pages, 33 figures; added appendix with additional language results

References in corpus (15)

• Neural Discrete Representation Learning
• Scaling Laws for Neural Language Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Generating Long Sequences with Sparse Transformers
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• An Empirical Model of Large-Batch Training
• Jukebox: A Generative Model for Music
• Analysing Mathematical Reasoning Abilities of Neural Models
• Generating Wikipedia by Summarizing Long Sequences
• The large learning rate phase of deep learning: the catapult mechanism
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• A Neural Scaling Law from the Dimension of the Data Manifold
• One Epoch Is All You Need
• On the Predictability of Pruning Across Scales
• Compositionality decomposed: how do neural networks generalise?

Cited by in corpus (40)

• On the Opportunities and Risks of Foundation Models
• Reproducible scaling laws for contrastive language-image learning
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Embodied Intelligence via Learning and Evolution
• Measuring Mathematical Problem Solving With the MATH Dataset
• Compute and Energy Consumption Trends in Deep Learning Inference
• Revisiting ResNets: Improved Training and Scaling Strategies
• Explaining Neural Scaling Laws
• Unsolved Problems in ML Safety
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Efficient shallow learning as an alternative to deep learning
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• Scaling Laws for Transfer
• A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
• Scaling Laws for Neural Machine Translation
• Scaling Up Vision-Language Pre-training for Image Captioning
• Why and How Governments Should Monitor AI Development
• Generalization bounds for deep learning
• Unsupervised Neural Machine Translation with Generative Language Models Only
• Designing lensless imaging systems to maximize information capture
• Scaling Scaling Laws with Board Games
• Learning Curve Theory
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
• Proof Artifact Co-training for Theorem Proving with Language Models
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
• 8-bit Optimizers via Block-wise Quantization
• High-Resolution Complex Scene Synthesis with Transformers
• DP-REC: Private & Communication-Efficient Federated Learning
• A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesis
• The Complexity Dynamics of Grokking
• Investigation of Training Label Error Impact on RNN-T
• Is the Number of Trainable Parameters All That Actually Matters?
• Universal Policies for Software-Defined MDPs
• Search Spaces for Neural Model Training
• Teaching Autoregressive Language Models Complex Tasks By Demonstration
• Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers
• Turing-Universal Learners with Optimal Scaling Laws