Theory of the Frequency Principle for General Deep Neural Networks

Abstract

Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.

CSIAM Trans. Appl. Math

References in corpus (9)

• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• ReLU Deep Neural Networks and Linear Finite Elements
• Multi-scale Convolutional Neural Networks for Inverse Problems
• Machine Learning from a Continuous Viewpoint
• A multi-scale DNN algorithm for nonlinear elliptic equations with multiple scales

Cited by in corpus (9)

• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Reproducing Activation Function for Deep Learning
• Inverse modified differential equations for discovery of dynamics
• Deep frequency principle towards understanding why deeper learning is faster
• Why Shallow Networks Struggle to Approximate and Learn High Frequencies
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
• Neural network initialization with nonlinear characteristics and information on hierarchical features
• Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning