The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies

Abstract

We study the relationship between the frequency of a function and the speed at which a neural network learns it. We build on recent results that show that the dynamics of overparameterized neural networks trained with gradient descent can be well approximated by a linear system. When normalized training data is uniformly distributed on a hypersphere, the eigenfunctions of this linear system are spherical harmonic functions. We derive the corresponding eigenvalues for each frequency after introducing a bias term in the model. This bias term had been omitted from the linear network model without significantly affecting previous theoretical results. However, we show theoretically and experimentally that a shallow neural network without bias cannot represent or learn simple, low frequency functions with odd frequencies. Our results lead to specific predictions of the time it will take a network to learn functions of varying frequency. These predictions match the empirical behavior of both shallow and deep networks.

References in corpus (15)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• On the Number of Linear Regions of Deep Neural Networks
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• On the Convergence Rate of Training Recurrent Neural Networks
• Gradient descent aligns the layers of deep linear networks
• Diverse Neural Network Learns True Target Functions
• Risk and parameter convergence of logistic regression
• Understanding training and generalization in deep learning by Fourier analysis
• Weight Sharing is Crucial to Succesful Optimization

Cited by in corpus (40)

• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
• When and why PINNs fail to train: A neural tangent kernel perspective
• Multilevel domain decomposition-based architectures for physics-informed neural networks
• Towards Understanding the Spectral Bias of Deep Learning
• Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Gradient Starvation: A Learning Proclivity in Neural Networks
• A Fine-Grained Spectral Perspective on Neural Networks
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• LOCA: LOcal Conformal Autoencoder for standardized data coordinates
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
• The Implicit Bias of Depth: How Incremental Learning Drives Generalization
• On the Similarity between the Laplace and Neural Tangent Kernels
• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• Implicit Regularization via Neural Feature Alignment
• Shift Invariance Can Reduce Adversarial Robustness
• Deep frequency principle towards understanding why deeper learning is faster
• Kernel-Based Smoothness Analysis of Residual Networks
• General Probabilistic Surface Optimization and Log Density Estimation
• Exploring The Effect of High-frequency Components in GANs Training
• On the Generalization Power of Overfitted Two-Layer Neural Tangent Kernel Models
• Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Flow field tomography with uncertainty quantification using a Bayesian physics-informed neural network
• Uniform Convergence, Adversarial Spheres and a Simple Remedy
• Learning curves for Gaussian process regression with power-law priors and targets
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks
• Deep Neural Networks as Point Estimates for Deep Gaussian Processes
• The staircase property: How hierarchical structure can guide deep learning
• Uniform Generalization Bounds for Overparameterized Neural Networks
• On the Frequency Bias of Generative Models
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data
• Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
• Transforming physics-informed machine learning to convex optimization