Deep frequency principle towards understanding why deeper learning is faster
arXiv:2007.14313
Abstract
Understanding the effect of depth in deep learning is a critical problem. In this work, we utilize the Fourier analysis to empirically provide a promising mechanism to understand why feedforward deeper learning is faster. To this end, we separate a deep neural network, trained by normal stochastic gradient descent, into two parts during analysis, i.e., a pre-condition component and a learning component, in which the output of the pre-condition one is the input of the learning one. We use a filtering method to characterize the frequency distribution of a high-dimensional function. Based on experiments of deep networks and real dataset, we propose a deep frequency principle, that is, the effective target function for a deeper hidden layer biases towards lower frequency during the training. Therefore, the learning component effectively learns a lower frequency function if the pre-condition component has more layers. Due to the well-studied frequency principle, i.e., deep neural networks learn lower frequency functions faster, the deep frequency principle provides a reasonable explanation to why deeper learning is faster. We believe these empirical studies would be valuable for future theoretical studies of the effect of depth in deep learning.
References in corpus (21)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- On Exact Computation with an Infinitely Wide Neural Net
- A Closer Look at Memorization in Deep Networks
- Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
- The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
- Deep learning generalizes because the parameter-function map is biased towards simple functions
- Theory of the Frequency Principle for General Deep Neural Networks
- Multi-scale Convolutional Neural Networks for Inverse Problems
- SGD on Neural Networks Learns Functions of Increasing Complexity
- Effect of Depth and Width on Local Minima in Deep Learning
- Understanding training and generalization in deep learning by Fourier analysis
- A Fine-Grained Spectral Perspective on Neural Networks
- Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
- Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
- A type of generalization error induced by initialization in deep neural networks
- Phase diagram for two-layer ReLU neural networks at infinite-width limit
- Neural networks are a priori biased towards Boolean functions with low entropy
- Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks