A Constructive Prediction of the Generalization Error Across Scales

Abstract

The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.

arXiv commentICLR 2020

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Wide Residual Networks
• Neural Architecture Search: A Survey
• Fine-Grained Visual Classification of Aircraft
• Learning multiple visual domains with residual adapters
• Pointer Sentinel Mixture Models
• Deep Learning Scaling is Predictable, Empirically
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Optimal approximation of continuous functions by very deep ReLU networks
• On the Convergence Rate of Training Recurrent Neural Networks
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Stronger generalization bounds for deep nets via a compression approach

Cited by in corpus (16)

• Scaling Laws for Neural Language Models
• Compute Trends Across Three Eras of Machine Learning
• Explaining Neural Scaling Laws
• A General Language Assistant as a Laboratory for Alignment
• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• Power-law Scaling to Assist with Key Challenges in Artificial Intelligence
• A Neural Scaling Law from the Dimension of the Data Manifold
• Robust Millimeter Beamforming via Self-Supervised Hybrid Deep Learning
• Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts
• Scaling Scaling Laws with Board Games
• Towards More Fine-grained and Reliable NLP Performance Prediction
• Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study
• Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images
• A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data
• Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
• Benchmarking down-scaled (not so large) pre-trained language models