On the Spectral Bias of Neural Networks

Abstract

Neural networks are known to be a class of highly expressive functions able to fit even random input-output mappings with $100\%$ accuracy. In this work, we present properties of neural networks that complement this aspect of expressivity. By using tools from Fourier analysis, we show that deep ReLU networks are biased towards low frequency functions, meaning that they cannot have local fluctuations without affecting their global behavior. Intuitively, this property is in line with the observation that over-parameterized networks find simple patterns that generalize across data samples. We also investigate how the shape of the data manifold affects expressivity by showing evidence that learning high frequencies gets \emph{easier} with increasing manifold complexity, and present a theoretical understanding of this behavior. Finally, we study the robustness of the frequency components with respect to parameter perturbation, to develop the intuition that the parameters must be finely tuned to express high frequency functions.

arXiv comment23 pages

Cited by in corpus (95)

• DeepXDE: A deep learning library for solving differential equations
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Adaptive activation functions accelerate convergence in deep and physics-informed neural networks
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• PINNeik: Eikonal solution using physics-informed neural networks
• Multi-scale Deep Neural Network (MscaleDNN) for Solving Poisson-Boltzmann Equation in Complex Domains
• Physics-informed neural networks for the shallow-water equations on the sphere
• When and why PINNs fail to train: A neural tangent kernel perspective
• Fighting deepfakes by detecting GAN DCT anomalies
• 3D Infomax improves GNNs for Molecular Property Prediction
• Towards Understanding the Spectral Bias of Deep Learning
• Object-Centric Neural Scene Rendering
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• On quadrature rules for solving Partial Differential Equations using Neural Networks
• Gradient Starvation: A Learning Proclivity in Neural Networks
• On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes
• UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution
• Explicit Regularisation in Gaussian Noise Injections
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Rethinking Positional Encoding
• Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior
• Focal Frequency Loss for Image Reconstruction and Synthesis
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks
• Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields
• SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization
• Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• SWAGAN: A Style-based Wavelet-driven Generative Model
• CoIL: Coordinate-based Internal Learning for Imaging Inverse Problems
• Unsupervised part representation by Flow Capsules
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime
• Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• SSD-GAN: Measuring the Realness in the Spatial and Spectral Domains
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• Neural Knitworks: Patched Neural Implicit Representation Networks
• Neural networks are a priori biased towards Boolean functions with low entropy
• Fourier Analysis-based Iterative Combinatorial Auctions
• Deep Point Cloud Reconstruction
• Frequency-Aware Reconstruction of Fluid Simulations with Generative Networks
• How Framelets Enhance Graph Neural Networks
• Is SGD a Bayesian sampler? Well, almost
• Predicting Infectiousness for Proactive Contact Tracing
• 2-Step Sparse-View CT Reconstruction with a Domain-Specific Perceptual Network
• COTR: Correspondence Transformer for Matching Across Images
• Positional Encoding as Spatial Inductive Bias in GANs
• Neural Anisotropy Directions
• Implicit Regularization via Neural Feature Alignment
• Physics Informed Deep Learning for Flow and Transport in Porous Media
• Shift Invariance Can Reduce Adversarial Robustness
• Frequency-compensated PINNs for Fluid-dynamic Design Problems
• Is it time to swish? Comparing activation functions in solving the Helmholtz equation using physics-informed neural networks
• Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
• Neural Fields in Visual Computing and Beyond
• Hierarchical Learning to Solve Partial Differential Equations Using Physics-Informed Neural Networks
• Spline Positional Encoding for Learning 3D Implicit Signed Distance Fields
• The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation
• Band-limited Soft Actor Critic Model
• Object Detection based on OcSaFPN in Aerial Images with Noise
• Data-informed Deep Optimization
• A neural anisotropic view of underspecification in deep learning
• MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis
• Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
• Which Minimizer Does My Neural Network Converge To?
• NeRV: Neural Representations for Videos
• ImplicitVol: Sensorless 3D Ultrasound Reconstruction with Deep Implicit Representation
• Initializing ReLU networks in an expressive subspace of weights
• What training reveals about neural network complexity
• Making Coherence Out of Nothing At All: Measuring the Evolution of Gradient Alignment
• Optimization Variance: Exploring Generalization Properties of DNNs
• PBNS: Physically Based Neural Simulator for Unsupervised Garment Pose Space Deformation
• Assessing the Bilingual Knowledge Learned by Neural Machine Translation Models
• Depth separation beyond radial functions
• Exploring The Effect of High-frequency Components in GANs Training
• Explaining generalization in deep learning: progress and fundamental limits
• Improving learnability of neural networks: adding supplementary axes to disentangle data representation
• Length Scale Control in Topology Optimization using Fourier Enhanced Neural Networks
• Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• PointMixer: MLP-Mixer for Point Cloud Understanding
• Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time
• Deep Autoencoders: From Understanding to Generalization Guarantees
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
• Parallel frequency function-deep neural network for efficient complex broadband signal approximation
• On the Frequency Bias of Generative Models
• Over-parametrized neural networks as under-determined linear systems
• Fourier-domain Variational Formulation and Its Well-posedness for Supervised Learning
• IREM: High-Resolution Magnetic Resonance (MR) Image Reconstruction via Implicit Neural Representation
• FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics
• A Frequency Domain Constraint for Synthetic and Real X-ray Image Super Resolution