Neural Fields in Visual Computing and Beyond

Abstract

Recent advances in machine learning have created increasing interest in solving visual computing problems using a class of coordinate-based neural networks that parametrize physical properties of scenes or objects across space and time. These methods, which we call neural fields, have seen successful application in the synthesis of 3D shapes and image, animation of human bodies, 3D reconstruction, and pose estimation. However, due to rapid progress in a short time, many papers exist but a comprehensive review and formulation of the problem has not yet emerged. In this report, we address this limitation by providing context, mathematical grounding, and an extensive review of literature on neural fields. This report covers research along two dimensions. In Part I, we focus on techniques in neural fields by identifying common components of neural field methods, including different representations, architectures, forward mapping, and generalization methods. In Part II, we focus on applications of neural fields to different problems in visual computing, and beyond (e.g., robotics, audio). Our review shows the breadth of topics already covered in visual computing, both historically and in current incarnations, demonstrating the improved quality, flexibility, and capability brought by neural fields methods. Finally, we present a companion website that contributes a living version of this review that can be continually updated by the community.

Equal advising: Vincent Sitzmann and Srinath Sridhar

References in corpus (37)

• Alias-Free Generative Adversarial Networks
• Finite Basis Physics-Informed Neural Networks (FBPINNs): a scalable domain decomposition approach for solving differential equations
• When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?
• Neural Reflectance Fields for Appearance Acquisition
• MetaSDF: Meta-learning Signed Distance Functions
• Learning to Infer Implicit Surfaces without 3D Supervision
• NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
• Object-Centric Neural Scene Rendering
• MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images
• Portrait Neural Radiance Fields from a Single Image
• Rethinking Positional Encoding
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model
• Fisher Information Field: an Efficient and Differentiable Map for Perception-aware Planning
• 3D Neural Scene Representations for Visuomotor Control
• Neural Trajectory Fields for Dynamic Novel View Synthesis
• PVA: Pixel-aligned Volumetric Avatars
• CoIL: Coordinate-based Internal Learning for Imaging Inverse Problems
• NeRF in detail: Learning to sample for view synthesis
• Neural Knitworks: Patched Neural Implicit Representation Networks
• The Whole Is Greater Than the Sum of Its Nonrigid Parts
• Volume Rendering of Neural Implicit Surfaces
• ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
• Neural 3D Scene Compression via Model Compression
• Implicit-PDF: Non-Parametric Representation of Probability Distributions on the Rotation Manifold
• Augmenting Implicit Neural Shape Representations with Explicit Deformation Fields
• Direct-PoseNet: Absolute Pose Regression with Photometric Consistency
• SALD: Sign Agnostic Learning with Derivatives
• Editing Conditional Radiance Fields
• Geodesy of irregular small bodies via neural density fields: geodesyNets
• NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering using RGB Cameras
• GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
• MirrorNeRF: One-shot Neural Portrait Radiance Field from Multi-mirror Catadioptric Imaging
• Modulated Periodic Activations for Generalizable Local Functional Representations
• Neural Image Representations for Multi-Image Fusion and Layer Separation
• Deep Medial Fields

Cited by in corpus (4)

• CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory
• CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation
• NSTO: Neural Synthesizing Topology Optimization for Modulated Structure Generation
• IntraSeismic: a coordinate-based learning approach to seismic inversion