Neural Scaling Laws for Deep Regression

Abstract

Neural scaling laws--power-law relationships between generalization errors and characteristics of deep learning models--are vital tools for developing reliable models while managing limited resources. Although the success of large language models highlights the importance of these laws, their application to deep regression models remains largely unexplored. Here, we empirically investigate neural scaling laws in deep regression using a parameter estimation model for twisted van der Waals magnets. We observe power-law relationships between the loss and both training dataset size and model capacity across a wide range of values, employing various architectures--including fully connected networks, residual networks, and vision transformers. Furthermore, the scaling exponents governing these relationships range from 1 to 2, with specific values depending on the regressed parameters and model details. The consistent scaling behaviors and their large scaling exponents suggest that the performance of deep regression models can improve substantially with increasing data size.

Supplementary Information will be provided with the published manuscript

References in corpus (20)

• Machine learning and the physical sciences
• Machine learning phases of matter
• A high-bias, low-variance introduction to Machine Learning for physicists
• Deep Learning and its Application to LHC Physics
• Discovering Phases, Phase Transitions and Crossovers through Unsupervised Machine Learning: A critical examination
• Topological spin excitations in honeycomb ferromagnet CrI${}_3$
• A Comprehensive Analysis of Deep Regression
• Fast Automated Analysis of Strong Gravitational Lenses with Convolutional Neural Networks
• Explaining Neural Scaling Laws
• Deep Learning the Quantum Phase Transitions in Random Two-Dimensional Electron Systems
• Machine Learning for Condensed Matter Physics
• Magnetization textures in twisted bilayer 2D CrX${}_3$ (X=Br, I)
• Drawing Phase Diagrams of Random Quantum Systems by Deep Learning the Wave Functions
• Magnetic skyrmion lattices in a novel two-dimensional twisted bilayer magnet
• Power-law Scaling to Assist with Key Challenges in Artificial Intelligence
• Emergence of stable meron quartets in twisted magnets
• Controllable magnetic domains in twisted trilayer magnets
• Machine learning of phases and structures for model systems in physics
• Deep learning methods for Hamiltonian parameter estimation and magnetic domain image generation in twisted van der Waals magnets
• Emergence of Meron Kekulé lattices in twisted Néel antiferromagnets