Uncertainty Quantification From Scaling Laws in Deep Neural Networks

Abstract

Quantifying the uncertainty from machine learning analyses is critical to their use in the physical sciences. In this work we focus on uncertainty inherited from the initialization distribution of neural networks. We compute the mean ${\mu }_{\mathcal{L}}$ and variance ${\sigma }_{\mathcal{L}}^2$ of the test loss $\mathcal{L}$ for an ensemble of multi-layer perceptrons (MLPs) with neural tangent kernel (NTK) initialization in the infinite-width limit, and compare empirically to the results from finite-width networks for three example tasks: MNIST classification, CIFAR classification and calorimeter energy regression. We observe scaling laws as a function of training set size $N_{\mathcal{D}}$ for both ${\mu }_{\mathcal{L}}$ and ${\sigma }_{\mathcal{L}}$, but find that the coefficient of variation ${\epsilon }_{\mathcal{L}}\equiv {\sigma }_{\mathcal{L}}/{\mu }_{\mathcal{L}}$ becomes independent of $N_{\mathcal{D}}$ at both infinite and finite width for sufficiently large $N_{\mathcal{D}}$. This implies that the coefficient of variation of a finite-width network may be approximated by its infinite-width value, and may in principle be calculable using finite-width perturbation theory.

18+3 pages, 6 figures

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Scaling Laws for Neural Language Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Neural Networks as Gaussian Processes
• Deep Exploration via Bootstrapped DQN
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Explaining Neural Scaling Laws
• Calorimetry with Deep Learning: Particle Simulation and Reconstruction for Collider Physics
• Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
• Gaussian Process Behaviour in Wide Deep Neural Networks
• Neural-net-induced Gaussian process regression for function approximation and PDE solution
• Deep-Learning Jets with Uncertainties and More
• Per-Object Systematics using Deep-Learned Calibration
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• A Solvable Model of Neural Scaling Laws
• Interpretable Uncertainty Quantification in AI for HEP
• Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
• A Dynamical Model of Neural Scaling Laws
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Kernel Regression with Infinite-Width Neural Networks on Millions of Examples
• How Feature Learning Can Improve Neural Scaling Laws