Towards Understanding the Spectral Bias of Deep Learning

Abstract

An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions. The priority of learning functions with low complexity might be at the core of explaining generalization ability of neural network, and certain efforts have been made to provide theoretical explanation for spectral bias. However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias. In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work. We prove that the training process of neural networks can be decomposed along different directions defined by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue. We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks. Finally, we provide numerical experiments to demonstrate the correctness of our theory. Our experimental results also show that our theory can tolerate certain model misspecification in terms of the input data distribution.

arXiv comment29 pages, 7 figures. This version adds more experimental results

References in corpus (28)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• On Exact Computation with an Infinitely Wide Neural Net
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• On Lazy Training in Differentiable Programming
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• On the Spectral Bias of Neural Networks
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• What Can ResNet Learn Efficiently, Going Beyond Kernels?
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
• SGD on Neural Networks Learns Functions of Increasing Complexity
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• A Fine-Grained Spectral Perspective on Neural Networks
• Understanding training and generalization in deep learning by Fourier analysis
• Diving into the shallows: a computational perspective on large-scale shallow learning
• Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Gradient Descent for One-Hidden-Layer Neural Networks: Polynomial Convergence and SQ Lower Bounds
• Approximation beats concentration? An approximation view on inference with smooth radial kernels
• Implicit Regularization of Random Feature Models
• Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS

Cited by in corpus (18)

• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Benign Overfitting in Linear Regression
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• Gradient Starvation: A Learning Proclivity in Neural Networks
• Spectral-wise Implicit Neural Representation for Hyperspectral Image Reconstruction
• Neural Thompson Sampling
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Implicit Regularization via Neural Feature Alignment
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• Universal scaling laws in the gradient descent training of neural networks
• ConDA: Unsupervised Domain Adaptation for LiDAR Segmentation via Regularized Domain Concatenation
• Exploring The Effect of High-frequency Components in GANs Training
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysis
• Neural Active Learning with Performance Guarantees