Compute Trends Across Three Eras of Machine Learning

Abstract

Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified factor - compute. We show that before 2010 training compute grew in line with Moore's law, doubling roughly every 20 months. Since the advent of Deep Learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months. In late 2015, a new trend emerged as firms developed large-scale ML models with 10 to 100-fold larger requirements in training compute. Based on these observations we split the history of compute in ML into three eras: the Pre Deep Learning Era, the Deep Learning Era and the Large-Scale Era. Overall, our work highlights the fast-growing compute requirements for training advanced ML systems.

References in corpus (37)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• Improving neural networks by preventing co-adaptation of feature detectors
• Denoising Diffusion Probabilistic Models
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Generative Adversarial Networks
• Neural Architecture Search with Reinforcement Learning
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• PaLM: Scaling Language Modeling with Pathways
• Scaling Laws for Neural Language Models
• Flamingo: a Visual Language Model for Few-Shot Learning
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker
• LaMDA: Language Models for Dialog Applications
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Training Compute-Optimal Large Language Models
• Deep Learning Scaling is Predictable, Empirically
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• CogView: Mastering Text-to-Image Generation via Transformers
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• The Computational Limits of Deep Learning
• The Hanabi Challenge: A New Frontier for AI Research
• Self-supervised Pretraining of Visual Features in the Wild
• The Cost of Training NLP Models: A Concise Overview
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• A Constructive Prediction of the Generalization Error Across Scales
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
• Scaling Scaling Laws with Board Games

Cited by in corpus (31)

• Experimentally realized in situ backpropagation for deep learning in nanophotonic neural networks
• How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Study
• Large Language Models and the Reverse Turing Test
• ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Review
• Quantum Machine Learning: from physics to software engineering
• Astronomia ex machina: a history, primer, and outlook on neural networks in astronomy
• Carbon Footprint of Selecting and Training Deep Learning Models for Medical Image Analysis
• Three lines of defense against risks from AI
• Trends in Energy Estimates for Computing in AI/Machine Learning Accelerators, Supercomputers, and Compute-Intensive Applications
• A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model
• Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
• Efficiency is Not Enough: A Critical Perspective of Environmentally Sustainable AI
• Ultrafast Coherent Dynamics of Microring Modulators
• EC-NAS: Energy Consumption Aware Tabular Benchmarks for Neural Architecture Search
• Rethinking model prototyping through the MedMNIST+ dataset collection
• A Green(er) World for A.I
• Operating critical machine learning models in resource constrained regimes
• Phase-space analysis of a two-section InP laser as an all-optical spiking neuron: dependency on control and design parameters
• Energy Efficiency trends in HPC: what high-energy and astrophysicists need to know
• A 262 TOPS Hyperdimensional Photonic AI Accelerator powered by a Si3N4 microcomb laser
• Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimization
• FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
• Experimental Standards for Deep Learning in Natural Language Processing Research
• A Reconfigurable Stream-Based FPGA Accelerator for Bayesian Confidence Propagation Neural Networks
• Dual-Issue Execution of Mixed Integer and Floating-Point Workloads on Energy-Efficient In-Order RISC-V Cores
• TeraPool: A Physical Design Aware, 1024 RISC-V Cores Shared-L1-Memory Scaled-up Cluster Design with High Bandwidth Main Memory Link
• A Memory Efficient Adjoint Method to Enable Billion Parameter Optimization on a Single GPU in Dynamic Problems
• Virgo: Cluster-level Matrix Unit Integration in GPUs for Scalability and Energy Efficiency
• How Small is Big Enough? Open Labeled Datasets and the Development of Deep Learning
• Accelerating Sharded Data Parallelism at Scale with Federated Learning
• Oxide Interface-Based Polymorphic Electronic Devices for Neuromorphic Computing