Deep Learning Scaling is Predictable, Empirically

Abstract

Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art. This paper presents a large scale empirical characterization of generalization error and model size growth as training sets grow. We introduce a methodology for this measurement and test four machine learning domains: machine translation, language modeling, image processing, and speech recognition. Our empirical results show power-law generalization error scaling across a breadth of factors, resulting in power-law exponents---the "steepness" of the learning curve---yet to be explained by theoretical work. Further, model improvements only shift the error but do not appear to affect the power-law exponent. We also show that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.

19 pages, 11 figures

References in corpus (5)

• Attention-Based Models for Speech Recognition
• Exploring the Limits of Language Modeling
• A Closer Look at Memorization in Deep Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Capacity and Trainability in Recurrent Neural Networks

Cited by in corpus (97)

• Learning Transferable Visual Models From Natural Language Supervision
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials
• Reproducible scaling laws for contrastive language-image learning
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Multimodal End-to-End Autonomous Driving
• Compute Trends Across Three Eras of Machine Learning
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• The Proper Care and Feeding of CAMELS: How Limited Training Data Affects Streamflow Prediction
• Compute and Energy Consumption Trends in Deep Learning Inference
• Revisiting ResNets: Improved Training and Scaling Strategies
• Deep-Learning-Based Image Segmentation Integrated with Optical Microscopy for Automatically Searching for Two-Dimensional Materials
• Measuring the Effects of Data Parallelism on Neural Network Training
• Explaining Neural Scaling Laws
• The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research
• Benchmarking of eight recurrent neural network variants for breath phase and adventitious sound detection on a self-developed open-access lung sound database-HF_Lung_V1
• Unsolved Problems in ML Safety
• Ground state energy functional with Hartree-Fock efficiency and chemical accuracy
• A factorisation-aware Matrix element emulator
• A Constructive Prediction of the Generalization Error Across Scales
• Data Valuation using Reinforcement Learning
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
• Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis
• Efficient shallow learning as an alternative to deep learning
• Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
• Cerebro: Static Subsuming Mutant Selection
• Is it enough to optimize CNN architectures on ImageNet?
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• Scaling Laws for Deep Learning
• How fine can fine-tuning be? Learning efficient language models
• GDP: Generalized Device Placement for Dataflow Graphs
• Mind the Gap: Federated Learning Broadens Domain Generalization in Diagnostic AI Models
• Power-law Scaling to Assist with Key Challenges in Artificial Intelligence
• Structured Model Pruning of Convolutional Networks on Tensor Processing Units
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Scaling Laws for Neural Machine Translation
• Using Error Decay Prediction to Overcome Practical Issues of Deep Active Learning for Named Entity Recognition
• Robust Millimeter Beamforming via Self-Supervised Hybrid Deep Learning
• One Epoch Is All You Need
• On the Predictability of Pruning Across Scales
• Machine learning surrogates for efficient hydrologic modeling: Insights from stochastic simulations of managed aquifer recharge
• Technical Considerations for Semantic Segmentation in MRI using Convolutional Neural Networks
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
• Why and How Governments Should Monitor AI Development
• Generalization in Machine Learning via Analytical Learning Theory
• Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records
• Generalization bounds for deep learning
• Network Support for High-performance Distributed Machine Learning
• The Low-Rank Simplicity Bias in Deep Networks
• Supervised learning of random quantum circuits via scalable neural networks
• Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality
• Learning Curves for Analysis of Deep Networks
• Data Leverage: A Framework for Empowering the Public in its Relationship with Technology Companies
• Is SGD a Bayesian sampler? Well, almost
• Engineering Reliable Deep Learning Systems
• Deep Learning is Singular, and That's Good
• Scaling Scaling Laws with Board Games
• Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
• Learning Curve Theory
• Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Minimizers of the Empirical Risk and Risk Monotonicity
• Embedded Large-Scale Handwritten Chinese Character Recognition
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• RankingMatch: Delving into Semi-Supervised Learning with Consistency Regularization and Ranking Loss
• An Update on a Progressively Expanded Database for Automated Lung Sound Analysis
• Untrained neural networks can demonstrate memorization-independent abstract reasoning
• Extractive Summary as Discrete Latent Variables
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• ExplainFix: Explainable Spatially Fixed Deep Networks
• Are wider nets better given the same number of parameters?
• Improving Differentially Private Models with Active Learning
• Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images
• Designing for the Long Tail of Machine Learning
• Is the Number of Trainable Parameters All That Actually Matters?
• Learning curves for Gaussian process regression with power-law priors and targets
• Search Spaces for Neural Model Training
• Efficient Client Contribution Evaluation for Horizontal Federated Learning
• Max and Coincidence Neurons in Neural Networks
• Top squark signal significance enhancement by different Machine Learning Algorithms
• Learning to Transfer Learn: Reinforcement Learning-Based Selection for Adaptive Transfer Learning
• Data Optimisation for a Deep Learning Recommender System
• Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers
• Turing-Universal Learners with Optimal Scaling Laws
• Slice Tuner: A Selective Data Acquisition Framework for Accurate and Fair Machine Learning Models
• A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data
• Leveraging GANs to Improve Continuous Path Keyboard Input Models
• The LLM Mirage: Economic Interests and the Subversion of Weaponization Controls
• Transferable Graph Optimizers for ML Compilers
• Challenging Images For Minds and Machines
• Robust Temporal Ensembling for Learning with Noisy Labels