Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Abstract

We show that passing input points through a simple Fourier feature mapping enables a multilayer perceptron (MLP) to learn high-frequency functions in low-dimensional problem domains. These results shed light on recent advances in computer vision and graphics that achieve state-of-the-art results by using MLPs to represent complex 3D objects and scenes. Using tools from the neural tangent kernel (NTK) literature, we show that a standard MLP fails to learn high frequencies both in theory and in practice. To overcome this spectral bias, we use a Fourier feature mapping to transform the effective NTK into a stationary kernel with a tunable bandwidth. We suggest an approach for selecting problem-specific Fourier features that greatly improves the performance of MLPs for low-dimensional regression tasks relevant to the computer vision and graphics communities.

Project page: https://people.eecs.berkeley.edu/~bmild/fourfeat/

References in corpus (6)

• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Learning to Infer Implicit Surfaces without 3D Supervision
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Time2Vec: Learning a Vector Representation of Time
• Self-attention with Functional Time Representation Learning
• Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation

Cited by in corpus (24)

• When and why PINNs fail to train: A neural tangent kernel perspective
• NAF: Neural Attenuation Fields for Sparse-View CBCT Reconstruction
• Physics-informed data based neural networks for two-dimensional turbulence
• Learning Specialized Activation Functions for Physics-informed Neural Networks
• NVIDIA SimNet^{TM}: an AI-accelerated multi-physics simulation framework
• Robust deformable image registration using cycle-consistent implicit representations
• Rethinking Positional Encoding
• Unsupervised part representation by Flow Capsules
• FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
• Neural Parameterization for Dynamic Human Head Editing
• Reproducing Activation Function for Deep Learning
• Distilling Style from Image Pairs for Global Forward and Inverse Tone Mapping
• Polynomial Neural Fields for Subband Decomposition and Manipulation
• Transformers for One-Shot Visual Imitation
• Physics Informed Deep Learning for Flow and Transport in Porous Media
• Moser Flow: Divergence-based Generative Modeling on Manifolds
• Neural Distortion Fields for Spatial Calibration of Wide Field-of-View Near-Eye Displays
• NSTO: Neural Synthesizing Topology Optimization for Modulated Structure Generation
• ExplainFix: Explainable Spatially Fixed Deep Networks
• Machine Learning for Mini-EUSO Telescope Data Analysis
• Convolutional Neural Opacity Radiance Fields
• Explainable Machine Learning: The Importance of a System-Centric Perspective
• Realistic Image Synthesis with Configurable 3D Scene Layouts
• IREM: High-Resolution Magnetic Resonance (MR) Image Reconstruction via Implicit Neural Representation