Power-law Scaling to Assist with Key Challenges in Artificial Intelligence

Abstract

Power-law scaling, a central concept in critical phenomena, is found to be useful in deep learning, where optimized test errors on handwritten digit examples converge as a power-law to zero with database size. For rapid decision making with one training epoch, each example is presented only once to the trained network, the power-law exponent increased with the number of hidden layers. For the largest dataset, the obtained test error was estimated to be in the proximity of state-of-the-art algorithms for large epoch numbers. Power-law scaling assists with key challenges found in current artificial intelligence applications and facilitates an a priori dataset size estimation to achieve a desired test accuracy. It establishes a benchmark for measuring training complexity and a quantitative hierarchy of machine learning tasks and algorithms.

arXiv comment30 pages, 5 figures

References in corpus (12)

• Statistical mechanics of complex networks
• Catastrophic cascade of failures in interdependent networks
• Self-similarity of complex networks
• Unified Scaling Law for Earthquakes
• Discovering Phase Transitions with Unsupervised Learning
• Deep Learning Scaling is Predictable, Empirically
• Deep Learning and its Application to LHC Physics
• Low Photon Count Phase Retrieval Using Deep Learning
• Enabling real-time multi-messenger astrophysics discoveries with deep learning
• Machine Learning in a data-limited regime: Augmenting experiments with synthetic data uncovers order in crumpled sheets
• Optimal Network Topology for Effective Collective Response
• A Constructive Prediction of the Generalization Error Across Scales

Cited by in corpus (7)

• Supervised Hebbian Learning
• Efficient shallow learning as an alternative to deep learning
• Dense Hebbian neural networks: a replica symmetric picture of supervised learning
• Brain inspired neuronal silencing mechanism to enable reliable sequence identification
• Role of Delay in Brain Dynamics
• Neural Scaling Laws for Deep Regression
• Supervised and Unsupervised protocols for hetero-associative neural networks