Neural Fields in Visual Computing and Beyond

Abstract

Recent advances in machine learning have created increasing interest in solving visual computing problems using a class of coordinate-based neural networks that parametrize physical properties of scenes or objects across space and time. These methods, which we call neural fields, have seen successful application in the synthesis of 3D shapes and image, animation of human bodies, 3D reconstruction, and pose estimation. However, due to rapid progress in a short time, many papers exist but a comprehensive review and formulation of the problem has not yet emerged. In this report, we address this limitation by providing context, mathematical grounding, and an extensive review of literature on neural fields. This report covers research along two dimensions. In Part I, we focus on techniques in neural fields by identifying common components of neural field methods, including different representations, architectures, forward mapping, and generalization methods. In Part II, we focus on applications of neural fields to different problems in visual computing, and beyond (e.g., robotics, audio). Our review shows the breadth of topics already covered in visual computing, both historically and in current incarnations, demonstrating the improved quality, flexibility, and capability brought by neural fields methods. Finally, we present a companion website that contributes a living version of this review that can be continually updated by the community.

arXiv commentEqual advising: Vincent Sitzmann and Srinath Sridhar

References in corpus (50)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Alias-Free Generative Adversarial Networks
• NeRF++: Analyzing and Improving Neural Radiance Fields
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• NeRF--: Neural Radiance Fields Without Known Camera Parameters
• Optimizing the Latent Space of Generative Networks
• EikoNet: Solving the Eikonal equation with Deep Neural Networks
• On the Spectral Bias of Neural Networks
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction
• Neural Reflectance Fields for Appearance Acquisition
• MetaSDF: Meta-learning Signed Distance Functions
• Learning to Infer Implicit Surfaces without 3D Supervision
• NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
• Controlling Neural Level Sets
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
• Object-Centric Neural Scene Rendering
• MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images
• Portrait Neural Radiance Fields from a Single Image
• On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes
• CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations
• Rethinking Positional Encoding
• Fisher Information Field: an Efficient and Differentiable Map for Perception-aware Planning
• Reconstructing continuous distributions of 3D protein structure from cryo-EM images
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model
• 3D Neural Scene Representations for Visuomotor Control
• Neural Trajectory Fields for Dynamic Novel View Synthesis
• PVA: Pixel-aligned Volumetric Avatars
• CoIL: Coordinate-based Internal Learning for Imaging Inverse Problems
• PlenOctrees for Real-time Rendering of Neural Radiance Fields
• NeRF in detail: Learning to sample for view synthesis
• Neural Knitworks: Patched Neural Implicit Representation Networks
• The Whole Is Greater Than the Sum of Its Nonrigid Parts
• Volume Rendering of Neural Implicit Surfaces
• ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
• Neural 3D Scene Compression via Model Compression
• Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
• Augmenting Implicit Neural Shape Representations with Explicit Deformation Fields
• Direct-PoseNet: Absolute Pose Regression with Photometric Consistency
• Editing Conditional Radiance Fields
• SALD: Sign Agnostic Learning with Derivatives
• Geodesy of irregular small bodies via neural density fields: geodesyNets
• NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras
• MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging
• GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
• Modulated Periodic Activations for Generalizable Local Functional Representations
• Neural Image Representations for Multi-Image Fusion and Layer Separation
• Deep Medial Fields

Cited by in corpus (4)

• CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
• CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation
• NSTO: Neural Synthesizing Topology Optimization for Modulated Structure Generation
• IntraSeismic: a coordinate-based learning approach to seismic inversion