Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
arXiv:2102.00200 · doi:10.1088/0256-307X/38/3/038701
Abstract
Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to explain this non-overfitting puzzle. Our linear frequency principle (LFP) model accounts for a key dynamical feature of NNs: they learn low frequencies first, irrespective of microscopic details. Theory based on our LFP model shows that low frequency dominance of target functions is the key condition for the non-overfitting of NNs and is verified by experiments. Furthermore, through an ideal two-layer NN, we unravel how detailed microscopic NN training dynamics statistically gives rise to a LFP model with quantitative prediction power.
arXiv commentto appear in Chinese Physics Letters
References in corpus (16)
- Machine learning and the physical sciences
- Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics
- Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
- A high-bias, low-variance introduction to Machine Learning for physicists
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Deep Learning and its Application to LHC Physics
- A Convolutional Neural Network Neutrino Event Classifier
- Approximating quantum many-body wave-functions using artificial neural networks
- Quantum Entanglement in Deep Learning Architectures
- Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
- A Priori Estimates of the Population Risk for Two-layer Neural Networks
- A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
- Multi-scale Deep Neural Network (MscaleDNN) Methods for Oscillatory Stokes Flows in Complex Domains
- A multi-scale DNN algorithm for nonlinear elliptic equations with multiple scales
- Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks