Scaling Laws for Deep Learning

Abstract

Running faster will only get you so far -- it is generally advisable to first understand where the roads lead, then get a car ... The renaissance of machine learning (ML) and deep learning (DL) over the last decade is accompanied by an unscalable computational cost, limiting its advancement and weighing on the field in practice. In this thesis we take a systematic approach to address the algorithmic and methodological limitations at the root of these costs. We first demonstrate that DL training and pruning are predictable and governed by scaling laws -- for state of the art models and tasks, spanning image classification and language modeling, as well as for state of the art model compression via iterative pruning. Predictability, via the establishment of these scaling laws, provides the path for principled design and trade-off reasoning, currently largely lacking in the field. We then continue to analyze the sources of the scaling laws, offering an approximation-theoretic view and showing through the exploration of a noiseless realizable case that DL is in fact dominated by error sources very far from the lower error limit. We conclude by building on the gained theoretical understanding of the scaling laws' origins. We present a conjectural path to eliminate one of the current dominant error sources -- through a data bandwidth limiting hypothesis and the introduction of Nyquist learners -- which can, in principle, reach the generalization error lower limit (e.g. 0 in the noiseless case), at finite dataset size.

arXiv commentPhD thesis

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Language Models are Few-Shot Learners
• Wide Residual Networks
• Scaling Laws for Neural Language Models
• Fine-Grained Visual Classification of Aircraft
• Pointer Sentinel Mixture Models
• The State of Sparsity in Deep Neural Networks
• Deep Learning Scaling is Predictable, Empirically
• How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Explaining Neural Scaling Laws
• Measuring the Algorithmic Efficiency of Neural Networks
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Stronger generalization bounds for deep nets via a compression approach
• A Neural Scaling Law from the Dimension of the Data Manifold