Measuring the Algorithmic Efficiency of Neural Networks

Abstract

Three factors drive the advance of AI: algorithmic innovation, data, and the amount of compute available for training. Algorithmic progress has traditionally been more difficult to quantify than compute and data. In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.

20 pages, 5 figures

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Playing Atari with Deep Reinforcement Learning
• Deep Residual Learning for Image Recognition
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Densely Connected Convolutional Networks
• Wide Residual Networks
• Going Deeper with Convolutions
• Dota 2 with Large Scale Deep Reinforcement Learning
• Aggregated Residual Transformations for Deep Neural Networks
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• When Will AI Exceed Human Performance? Evidence from AI Experts
• Multi-Task Deep Neural Networks for Natural Language Understanding
• An Empirical Model of Large-Batch Training
• Fixup Initialization: Residual Learning Without Normalization

Cited by in corpus (18)

• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Compute and Energy Consumption Trends in Deep Learning Inference
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Carbon Emissions and Large Neural Network Training
• Towards Sustainable Artificial Intelligence: An Overview of Environmental Protection Uses and Issues
• Large image datasets: A pyrrhic win for computer vision?
• Scaling Laws for Transfer
• Scaling Laws for Deep Learning
• Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learning
• Efficient and Generic 1D Dilated Convolution Layer for Deep Learning
• Learned Hardware/Software Co-Design of Neural Accelerators
• Automated Essay Scoring Using Transformer Models
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Wasserstein Embedding for Graph Learning
• Greenformer: Factorization Toolkit for Efficient Deep Neural Networks
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Neural network scoring for efficient computing
• On Inductive Biases for Machine Learning in Data Constrained Settings