Scaling Laws for Neural Language Models

Abstract

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

19 pages, 15 figures

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Wide Residual Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Generating Long Sequences with Sparse Transformers
• Universal Transformers
• Scaling description of generalization with number of parameters in deep learning
• Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
• Entropy and Long range correlations in literary English
• Entropy and Long range correlations in literary English
• Measuring the Effects of Data Parallelism on Neural Network Training
• An Empirical Model of Large-Batch Training
• Gradient Descent Happens in a Tiny Subspace
• An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
• Mesh-TensorFlow: Deep Learning for Supercomputers
• A Constructive Prediction of the Generalization Error Across Scales
• One Epoch Is All You Need
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks

Cited by in corpus (287)

• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Evaluating Large Language Models Trained on Code
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Contrastive Representation Learning: A Framework and Review
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Beyond English-Centric Multilingual Machine Translation
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Towards Universal Neural Network Potential for Material Discovery Applicable to Arbitrary Combination of 45 Elements
• Computational bioacoustics with deep learning: a review and roadmap
• Improved Denoising Diffusion Probabilistic Models
• Practical Program Repair in the Era of Large Pre-trained Language Models
• LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Embodied Intelligence via Learning and Evolution
• Compute Trends Across Three Eras of Machine Learning
• Measuring Massive Multitask Language Understanding
• Measuring Mathematical Problem Solving With the MATH Dataset
• Extracting Training Data from Large Language Models
• High-Performance Large-Scale Image Recognition Without Normalization
• The Principles of Deep Learning Theory
• Compute and Energy Consumption Trends in Deep Learning Inference
• Auditing large language models: a three-layered approach
• Revisiting ResNets: Improved Training and Scaling Strategies
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Uncovering ChatGPT's Capabilities in Recommender Systems
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Scaling Laws for Autoregressive Generative Modeling
• Explaining Neural Scaling Laws
• Swin Transformer V2: Scaling Up Capacity and Resolution
• Large Language Models as Zero-Shot Conversational Recommenders
• Perceiver: General Perception with Iterative Attention
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• Harms from Increasingly Agentic Algorithmic Systems
• The Cost of Training NLP Models: A Concise Overview
• Pretrained Transformers for Text Ranking: BERT and Beyond
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
• Pre-training via Paraphrasing
• KLUE: Korean Language Understanding Evaluation
• SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network
• Large Language Models Can Be Strong Differentially Private Learners
• Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
• Transflower: probabilistic autoregressive dance generation with multimodal attention
• Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering
• A factorisation-aware Matrix element emulator
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
• Linguistically inspired roadmap for building biologically reliable protein language models
• GPT-4 can pass the Korean National Licensing Examination for Korean Medicine Doctors
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Unreflected Acceptance -- Investigating the Negative Consequences of ChatGPT-Assisted Problem Solving in Physics Education
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• Hash Layers For Large Sparse Models
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers
• Generating Bug-Fixes Using Pretrained Transformers
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Knowledge-Aware Language Model Pretraining
• Imitating Interactive Intelligence
• Enhanced Automated Code Vulnerability Repair using Large Language Models
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
• Nebula: Self-Attention for Dynamic Malware Analysis
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training
• FastMoE: A Fast Mixture-of-Expert Training System
• FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
• Achieving Peak Performance for Large Language Models: A Systematic Review
• Exploring the Limits of Large Scale Pre-training
• OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
• Pay Attention to MLPs
• Scalable and Efficient MoE Training for Multitask Multilingual Models
• Pretraining Representations for Data-Efficient Reinforcement Learning
• Several categories of Large Language Models (LLMs): A Short Survey
• Is it enough to optimize CNN architectures on ImageNet?
• How Large Language Models are Transforming Machine-Paraphrased Plagiarism
• Inseq: An Interpretability Toolkit for Sequence Generation Models
• Neural Symbolic Regression that Scales
• A General Language Assistant as a Laboratory for Alignment
• Large language models for automated scholarly paper review: A survey
• Predictive Coding: a Theoretical and Experimental Review
• How quantum computing can enhance biomarker discovery
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Scaling Laws for Transfer
• Scaling Laws for Deep Learning
• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• Large Knowledge Model: Perspectives and Challenges
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
• Statistical Query Algorithms and Low-Degree Tests Are Almost Equivalent
• Mind the Gap: Assessing Temporal Generalization in Neural Language Models
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• AI Challenges for Society and Ethics
• Intelligent Computing Social Modeling and Methodological Innovations in Political Science in the Era of Large Language Models
• Training Verifiers to Solve Math Word Problems
• Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT
• Neural Language Generation: Formulation, Methods, and Evaluation
• A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
• Revisiting Model Stitching to Compare Neural Representations
• Codified audio language modeling learns useful representations for music information retrieval
• Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Effect of Pre-Training Scale on Intra- and Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images
• Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
• Multimodal Large Language Models for Bioimage Analysis
• Variational operator learning: A unified paradigm marrying training neural operators and solving partial differential equations
• A Neural Scaling Law from the Dimension of the Data Manifold
• Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
• Spiers Memorial Lecture: How to do impactful research in artificial intelligence for chemistry and materials science
• Scaling Laws for Neural Machine Translation
• ReCycle: Resilient Training of Large DNNs using Pipeline Adaptation
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• What Matters for Adversarial Imitation Learning?
• What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
• Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
• The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Models
• Document Ranking with a Pretrained Sequence-to-Sequence Model
• Efficient Transformers with Dynamic Token Pooling
• HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems
• On the Predictability of Pruning Across Scales
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
• On Power Laws in Deep Ensembles
• Scaling Up Vision-Language Pre-training for Image Captioning
• The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers
• How to decay your learning rate
• Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
• Why and How Governments Should Monitor AI Development
• Pre-Trained Models: Past, Present and Future
• Large Language Models at Work in China's Labor Market
• M6-T: Exploring Sparse Expert Models and Beyond
• Supersonic: Learning to Generate Source Code Optimizations in C/C++
• Curriculum learning for language modeling
• Multi-step retrieval and reasoning improves radiology question answering with large language models
• Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts
• NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• Improving Automatic Parallel Training via Balanced Memory Workload Optimization
• Generalization bounds for deep learning
• Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning
• Exploring the Landscape of Natural Language Processing Research
• Semantic Edge Computing and Semantic Communications in 6G Networks: A Unifying Survey and Research Challenges
• The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints
• What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure
• Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
• Rethinking embedding coupling in pre-trained language models
• Exploring and Evaluating Personalized Models for Code Generation
• Primer: Searching for Efficient Transformers for Language Modeling
• AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
• The Low-Rank Simplicity Bias in Deep Networks
• Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain
• Eight challenges in developing theory of intelligence
• Transforming Agency. On the mode of existence of Large Language Models
• Agents for self-driving laboratories applied to quantum computing
• Generative artificial intelligence in dentistry: Current approaches and future challenges
• Self-Supervised Learning to Prove Equivalence Between Straight-Line Programs via Rewrite Rules
• Unsupervised Neural Machine Translation with Generative Language Models Only
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Truthful AI: Developing and governing AI that does not lie
• Parallel Training of Deep Networks with Local Updates
• Learning Curves for Analysis of Deep Networks
• Evaluating representations by the complexity of learning low-loss predictors
• Generating Images with Sparse Representations
• A recent evaluation on the performance of LLMs on radiation oncology physics using questions of randomly shuffled options
• Automated Essay Scoring Using Transformer Models
• Scaling Scaling Laws with Board Games
• Normative Conflicts and Shallow AI Alignment
• A-Eval: A Benchmark for Cross-Dataset Evaluation of Abdominal Multi-Organ Segmentation
• Deep Learning is Singular, and That's Good
• Is SGD a Bayesian sampler? Well, almost
• One4all User Representation for Recommender Systems in E-commerce
• Transformers with Competitive Ensembles of Independent Mechanisms
• ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
• Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data
• CNNs Avoid Curse of Dimensionality by Learning on Patches
• Learning Curve Theory
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• LLMs as Models for Analogical Reasoning
• Meta-Learning Initializations for Image Segmentation
• LIBRA: Enabling Workload-aware Multi-dimensional Network Topology Optimization for Distributed Training of Large AI Models
• Robust Optimization for Multilingual Translation with Imbalanced Data
• Arithmetic with Language Models: from Memorization to Computation
• Energy-information trade-off makes the cortical critical power law the optimal coding
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
• Tensor network to learn the wavefunction of data
• Application-Oriented Benchmarking of Quantum Generative Learning Using QUARK
• Language Modeling with Reduced Densities
• Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers
• SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
• Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks
• Insights into resource utilization of code small language models serving with runtime engines and execution providers
• Update Frequently, Update Fast: Retraining Semantic Parsing Systems in a Fraction of Time
• Which *BERT? A Survey Organizing Contextualized Encoders
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• ARIM-mdx Data System: Towards a Nationwide Data Platform for Materials Science
• A Closer Look at Codistillation for Distributed Training
• What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers
• A Generalizable Approach to Learning Optimizers
• A Generalized Lottery Ticket Hypothesis
• Wide Neural Networks Forget Less Catastrophically
• 8-bit Optimizers via Block-wise Quantization
• Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
• Visual Grounding Strategies for Text-Only Natural Language Processing
• Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?
• Stochastic Optimization with Laggard Data Pipelines
• Distributed Deep Learning in Open Collaborations
• Ambitions for theory in the physics of life
• Training a First-Order Theorem Prover from Synthetic Data
• Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
• K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining
• Exploring Large Language Models (LLMs) through interactive Python activities
• Bayesian RG Flow in Neural Network Field Theories
• Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study
• DP-REC: Private & Communication-Efficient Federated Learning
• DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• Layered gradient accumulation and modular pipeline parallelism: fast and efficient training of large language models
• High-Resolution Complex Scene Synthesis with Transformers
• Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
• Multi-scale Transformer Language Models
• Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model
• Visio-Verbal Teleimpedance Interface: Enabling Semi-Autonomous Control of Physical Interaction via Eye Tracking and Speech
• Active multi-fidelity Bayesian online changepoint detection
• Disentangling the Roles of Curation, Data-Augmentation and the Prior in the Cold Posterior Effect
• Towards More Effective and Economic Sparsely-Activated Model
• Towards Understanding the Benefits of Neural Network Parameterizations in Geophysical Inversions: A Study With Neural Fields
• BERT Goes Shopping: Comparing Distributional Models for Product Representations
• Current Limitations of Language Models: What You Need is Retrieval
• Speech Intention Understanding in a Head-final Language: A Disambiguation Utilizing Intonation-dependency
• The Complexity Dynamics of Grokking
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Roadmap to a Roadmap: How Could We Tell When AGI is a 'Manhattan Project' Away?
• Unsupervised detection of semantic correlations in big data
• Are wider nets better given the same number of parameters?
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• Uncertainty Quantification From Scaling Laws in Deep Neural Networks
• Scaling Law for Recommendation Models: Towards General-purpose User Representations
• Lower Perplexity is Not Always Human-Like
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• No News is Good News: A Critique of the One Billion Word Benchmark
• Out-of-distribution data supervision towards biomedical semantic segmentation
• Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs
• DAG Card is the new Model Card
• Universal Policies for Software-Defined MDPs
• Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark
• Investigation of Training Label Error Impact on RNN-T
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Search Spaces for Neural Model Training
• Compressed Communication for Distributed Training: Adaptive Methods and System
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building
• Mini Minds: Exploring Bebeshka and Zlata Baby Models
• INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers
• Why is AI hard and Physics simple?
• LLM-GROP: Visually Grounded Robot Task and Motion Planning with Large Language Models
• Artificial intelligence for representing and characterizing quantum systems
• CosmoGLINT: Cosmological Generative Model for Line Intensity Mapping with Transformer
• On a Class of Markov Order Estimators Based on PPM and Other Universal Codes
• LM-Critic: Language Models for Unsupervised Grammatical Error Correction
• Is the Number of Trainable Parameters All That Actually Matters?
• A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data
• Confidence Preservation Property in Knowledge Distillation Abstractions
• Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers
• On Anytime Learning at Macroscale
• Vectorizing string entries for data processing on tables: when are larger language models better?
• Universal Scale Laws for Colors and Patterns in Imagery
• Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation
• The LLM Mirage: Economic Interests and the Subversion of Weaponization Controls
• Revisiting Simple Neural Probabilistic Language Models
• Bounds for Algorithmic Mutual Information and a Unifilar Order Estimator
• Turing-Universal Learners with Optimal Scaling Laws
• Learning Size and Shape of Calabi-Yau Spaces
• Benchmarking down-scaled (not so large) pre-trained language models
• Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
• Emergent inabilities? Inverse scaling over the course of pretraining
• On Inductive Biases for Machine Learning in Data Constrained Settings
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units
• Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
• Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach
• Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets
• Fusing Sequence Motifs and Pan-Genomic Features: Antimicrobial Resistance Prediction using an Explainable Lightweight 1D CNN-XGBoost Ensemble
• The Efficiency Misnomer