Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Abstract

We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies -- on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

Paper is published in Communications in Computational Physics

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Understanding deep learning requires rethinking generalization
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• A Closer Look at Memorization in Deep Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Exploring Generalization in Deep Learning
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Uniformly Accurate Machine Learning Based Hydrodynamic Models for Kinetic Equations
• Data-Driven, Physics-Based Feature Extraction from Fluid Flow Fields
• Theory of the Frequency Principle for General Deep Neural Networks
• Failures of Gradient-Based Deep Learning
• Machine Learning from a Continuous Viewpoint
• Training behavior of deep neural network in frequency domain
• Understanding training and generalization in deep learning by Fourier analysis
• A multiscale neural network based on hierarchical matrices
• Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application
• Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks

Cited by in corpus (85)

• DeepXDE: A deep learning library for solving differential equations
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• When and why PINNs fail to train: A neural tangent kernel perspective
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• Fredformer: Frequency Debiased Transformer for Time Series Forecasting
• Multilevel domain decomposition-based architectures for physics-informed neural networks
• PINNup: Robust neural network wavefield solutions using frequency upscaling and neuron splitting
• Theory of the Frequency Principle for General Deep Neural Networks
• Towards Understanding the Spectral Bias of Deep Learning
• Multi-scale Deep Neural Network (MscaleDNN) Methods for Oscillatory Stokes Flows in Complex Domains
• Self-Supervised Coordinate Projection Network for Sparse-View Computed Tomography
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Training behavior of deep neural network in frequency domain
• Gradient Starvation: A Learning Proclivity in Neural Networks
• A Fine-Grained Spectral Perspective on Neural Networks
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• Deep learning phase recovery: data-driven, physics-driven, or combining both?
• MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Focal Frequency Loss for Image Reconstruction and Synthesis
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Continuous conditional generative adversarial networks for data-driven solutions of poroelasticity with heterogeneous material properties
• Learning Closed-form Equations for Subgrid-scale Closures from High-fidelity Data: Promises and Challenges
• The Spectral Bias of the Deep Image Prior
• Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks
• Backdoor Attack through Frequency Domain
• Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
• Understanding Deep Learning via Decision Boundary
• Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
• Physics-informed neural network for acoustic resonance analysis in a one-dimensional acoustic tube
• A Deep Neural Network for Knot Placement in B-spline Approximation
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• A Practitioner's Guide to Kolmogorov-Arnold Networks
• Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks
• Multiscale Physics-Informed Neural Networks for the Inverse Design of Hyperuniform Optical Materials
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• Reproducing Activation Function for Deep Learning
• Spatiotemporal Implicit Neural Representation as a Generalized Traffic Data Learner
• Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• Towards Understanding the Condensation of Neural Networks at Initial Training
• Conditional Density Estimation, Latent Variable Discovery and Optimal Transport
• A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems
• PhaseDNN - A Parallel Phase Shift Deep Neural Network for Adaptive Wideband Learning
• Convergence Analysis of Probability Flow ODE for Score-based Generative Models
• Deep frequency principle towards understanding why deeper learning is faster
• Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks
• The Slow Deterioration of the Generalization Error of the Random Feature Model
• Band-limited Soft Actor Critic Model
• Implicit bias with Ritz-Galerkin method in understanding deep learning for solving PDEs
• Physics-informed Partitioned Coupled Neural Operator for Complex Networks
• Why flatness does and does not correlate with generalization for deep neural networks
• Accurate and Robust Deep Learning Framework for Solving Wave-Based Inverse Problems in the Super-Resolution Regime
• Generic Neural Architecture Search via Regression
• Data-informed Deep Optimization
• A Mathematical Framework for Learning Probability Distributions
• Assessment of the Reliablity of a Model's Decision by Generalizing Attribution to the Wavelet Domain
• Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
• A neuron-wise subspace correction method for the finite neuron method
• Asymptotic-Preserving Neural Networks for Multiscale Time-Dependent Linear Transport Equations
• Why Shallow Networks Struggle to Approximate and Learn High Frequencies
• Exploring The Effect of High-frequency Components in GANs Training
• Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
• Deep learning-based computational method for soft matter dynamics: Deep Onsager-Machlup method
• Adaptive Learning on the Grids for Elliptic Hemivariational Inequalities
• Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
• Improving the Deconvolution of Spectrum at Finite Temperature via Neural Network
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Deep Learning without Global Optimization by Random Fourier Neural Networks
• Gated X-TFC: Soft Domain Decomposition for Forward and Inverse Problems in Sharp-Gradient PDEs
• A priori generalization error for two-layer ReLU neural network through minimum norm solution
• Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks
• On the Frequency Bias of Generative Models
• Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
• Robust Text CAPTCHAs Using Adversarial Examples
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
• Efficient Evaluation of Gravitational Lensing Amplification Factors: A Deep Learning Framework
• Understanding the Initial Condensation of Convolutional Neural Networks
• Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysis
• Parallel frequency function-deep neural network for efficient complex broadband signal approximation