An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network

Abstract

Deep neural network (DNN) usually learns the target function from low to high frequency, which is called frequency principle or spectral bias. This frequency principle sheds light on a high-frequency curse of DNNs -- difficult to learn high-frequency information. Inspired by the frequency principle, a series of works are devoted to develop algorithms for overcoming the high-frequency curse. A natural question arises: what is the upper limit of the decaying rate w.r.t. frequency when one trains a DNN? In this work, our theory, confirmed by numerical experiments, suggests that there is a critical decaying rate w.r.t. frequency in DNN training. Below the upper limit of the decaying rate, the DNN interpolates the training data by a function with a certain regularity. However, above the upper limit, the DNN interpolates the training data by a trivial function, i.e., a function is only non-zero at training data points. Our results indicate a better way to overcome the high-frequency curse is to design a proper pre-condition approach to shift high-frequency information to low-frequency one, which coincides with several previous developed algorithms for fast learning high-frequency information. More importantly, this work rigorously proves that the high-frequency curse is an intrinsic difficulty of DNNs.

arXiv commentarXiv admin note: substantial text overlap with arXiv:2012.03238

References in corpus (22)

• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Characterizing Implicit Bias in Terms of Optimization Geometry
• On the Spectral Bias of Neural Networks
• Neural Additive Models: Interpretable Machine Learning with Neural Nets
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Towards Understanding the Spectral Bias of Deep Learning
• Understanding training and generalization in deep learning by Fourier analysis
• A Fine-Grained Spectral Perspective on Neural Networks
• Implicit Bias of Gradient Descent on Linear Convolutional Networks
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• On the Implicit Bias of Dropout
• Accelerating Physics-Informed Neural Network Training with Prior Dictionaries
• A type of generalization error induced by initialization in deep neural networks
• Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
• Phase diagram for two-layer ReLU neural networks at infinite-width limit
• Learned Initializations for Optimizing Coordinate-Based Neural Representations
• Reproducing Activation Function for Deep Learning
• Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks
• Band-limited Soft Actor Critic Model