Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Abstract

We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.

arXiv commentProject page: https://people.eecs.berkeley.edu/~bmild/fourfeat/

References in corpus (16)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On the Spectral Bias of Neural Networks
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Learning to Infer Implicit Surfaces without 3D Supervision
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Time2Vec: Learning a Vector Representation of Time
• A Fine-Grained Spectral Perspective on Neural Networks
• Self-attention with Functional Time Representation Learning
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
• Reconstructing continuous distributions of 3D protein structure from cryo-EM images
• Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation
• NASA: Neural Articulated Shape Approximation

Cited by in corpus (125)

• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• NeRF++: Analyzing and Improving Neural Radiance Fields
• Segment Anything Model for Medical Images?
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination
• Variational Diffusion Models
• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
• When and why PINNs fail to train: A neural tangent kernel perspective
• Physics-informed neural networks of the Saint-Venant equations for downscaling a large-scale river model
• NeRF2: Neural Radio-Frequency Radiance Fields
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
• NAF: Neural Attenuation Fields for Sparse-View CBCT Reconstruction
• A survey on GANs for computer vision: Recent research, analysis and taxonomy
• Multilevel domain decomposition-based architectures for physics-informed neural networks
• Implicit Full Waveform Inversion with Deep Neural Representation
• Learning in Sinusoidal Spaces with Physics-Informed Neural Networks
• A novel meta-learning initialization method for physics-informed neural networks
• Enhanced physics-informed neural networks for hyperelasticity
• Continuous Remote Sensing Image Super-Resolution based on Context Interaction in Implicit Function Space
• Physics-informed data based neural networks for two-dimensional turbulence
• GIFS: Neural Implicit Function for General Shape Representation
• Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
• $\Delta$-PINNs: physics-informed neural networks on complex geometries
• FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos
• Self-Supervised Coordinate Projection Network for Sparse-View Computed Tomography
• Discovering interpretable elastoplasticity models via the neural polynomial method enabled symbolic regressions
• Spectral-wise Implicit Neural Representation for Hyperspectral Image Reconstruction
• DiNNO: Distributed Neural Network Optimization for Multi-Robot Collaborative Learning
• Learning Lattice Quantum Field Theories with Equivariant Continuous Flows
• Aero-Nef: Neural Fields for Rapid Aircraft Aerodynamics Simulations
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling
• WarpPINN: Cine-MR image registration with physics-informed neural networks
• Improved Training of Physics-Informed Neural Networks with Model Ensembles
• MOD-Net: A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs
• Robust deformable image registration using cycle-consistent implicit representations
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Rethinking Positional Encoding
• Random-Access Neural Compression of Material Textures
• Artemis: Articulated Neural Pets with Appearance and Motion synthesis
• Scalable Spatiotemporal Prediction with Bayesian Neural Fields
• CEENs: Causality-enforced evolutional networks for solving time-dependent partial differential equations
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Self-adaptive weights based on balanced residual decay rate for physics-informed neural networks and deep operator networks
• Blind Image Deblurring with Unknown Kernel Size and Substantial Noise
• Mapping Dark Matter in the Milky Way using Normalizing Flows and Gaia DR3
• Pansharpening via Frequency-Aware Fusion Network with Explicit Similarity Constraints
• Dynamic Cone-beam CT Reconstruction using Spatial and Temporal Implicit Neural Representation Learning (STINR)
• NASA: Neural Articulated Shape Approximation
• PaveSAM Segment Anything for Pavement Distress
• Remote Sensing Novel View Synthesis with Implicit Multiplane Representations
• Unsupervised part representation by Flow Capsules
• Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview
• FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
• MOISST: Multimodal Optimization of Implicit Scene for SpatioTemporal calibration
• SINR: Deconvolving Circular SAS Images Using Implicit Neural Representations
• 3DGR-CT: Sparse-View CT Reconstruction with a 3D Gaussian Representation
• Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields
• k-space Physics-informed Neural Network (k-PINN) for Compressed Spectral Mapping and Efficient Inversion of Vibrations in Thin Composite Laminates
• Physics-informed neural network for acoustic resonance analysis in a one-dimensional acoustic tube
• Operator Learning with Neural Fields: Tackling PDEs on General Geometries
• Event Transformer+. A multi-purpose solution for efficient event data processing
• Continuous PDE Dynamics Forecasting with Implicit Neural Representations
• SNeRV: Spectra-preserving Neural Representation for Video
• CKConv: Continuous Kernel Convolution For Sequential Data
• Generative Models as Distributions of Functions
• Coordinate-Based Seismic Interpolation in Irregular Land Survey: A Deep Internal Learning Approach
• Generative modeling of living cells with SO(3)-equivariant implicit neural representations
• Neural Parameterization for Dynamic Human Head Editing
• Neural Knitworks: Patched Neural Implicit Representation Networks
• Dynamic Structured Illumination Microscopy with a Neural Space-time Model
• Reproducing Activation Function for Deep Learning
• A Memory-Efficient Dynamic Image Reconstruction Method using Neural Fields
• Geometric implicit neural representations for signed distance functions
• Distilling Style from Image Pairs for Global Forward and Inverse Tone Mapping
• IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wild
• Diffusion-Based Audio Inpainting
• Spatiotemporal Implicit Neural Representation as a Generalized Traffic Data Learner
• MARF: The Medial Atom Ray Field Object Representation
• Composite Bayesian Optimization In Function Spaces Using NEON -- Neural Epistemic Operator Networks
• Teaching Robots to Build Simulations of Themselves
• Unsupervised neural-implicit laser absorption tomography for quantitative imaging of unsteady flames
• Polynomial Neural Fields for Subband Decomposition and Manipulation
• Neural 3D Scene Compression via Model Compression
• Rethinking Implicit Neural Representations for Vision Learners
• Transformers for One-Shot Visual Imitation
• On Infinite-Width Hypernetworks
• A Novel Implicit Neural Representation for Volume Data
• Physics-Informed Neural Networks for Transonic Flows around an Airfoil
• Path Integral Sampler: a stochastic control approach for sampling
• Predicting Change, Not States: An Alternate Framework for Neural PDE Surrogates
• Physics Informed Deep Learning for Flow and Transport in Porous Media
• Neural Distortion Fields for Spatial Calibration of Wide Field-of-View Near-Eye Displays
• On-the-go Reflectance Transformation Imaging with Ordinary Smartphones
• Neural Field Convolutions by Repeated Differentiation
• Moser Flow: Divergence-based Generative Modeling on Manifolds
• An operator preconditioning perspective on training in physics-informed machine learning
• Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models
• Scaling Up Machine Learning For Quantum Field Theory with Equivariant Continuous Flows
• Latent assimilation with implicit neural representations for unknown dynamics
• Global Adaptive Filtering Layer for Computer Vision
• NSTO: Neural Synthesizing Topology Optimization for Modulated Structure Generation
• Neural BSSRDF: Object Appearance Representation Including Heterogeneous Subsurface Scattering
• IntraSeismic: a coordinate-based learning approach to seismic inversion
• Normal-guided Detail-Preserving Neural Implicit Function for High-Fidelity 3D Surface Reconstruction
• Deep Neural Implicit Representation of Accessibility for Multi-Axis Manufacturing
• Density Ratio Estimation via Infinitesimal Classification
• Superpixel-informed Implicit Neural Representation for Multi-Dimensional Data
• ExplainFix: Explainable Spatially Fixed Deep Networks
• Time Series Continuous Modeling for Imputation and Forecasting with Implicit Neural Representations
• Machine Learning for Mini-EUSO Telescope Data Analysis
• DNN-Based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel
• Towards Understanding the Benefits of Neural Network Parameterizations in Geophysical Inversions: A Study With Neural Fields
• Fokker-Planck Score Learning: Efficient Free-Energy Estimation under Periodic Boundary Conditions
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Pattern Guided UV Recovery for Realistic Video Garment Texturing
• Explainable Machine Learning: The Importance of a System-Centric Perspective
• GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation
• Continuous Generative Neural Networks: A Wavelet-Based Architecture in Function Spaces
• Realistic Image Synthesis with Configurable 3D Scene Layouts
• Convolutional Neural Opacity Radiance Fields
• Adaptive Interpolating Quantum Transform: A Quantum-Native Framework for Efficient Transform Learning
• Spline-based Transformers
• Neural Computed Tomography
• IREM: High-Resolution Magnetic Resonance (MR) Image Reconstruction via Implicit Neural Representation