HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

Abstract

Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous. We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this "hyper-space". Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between "moments", i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS. Additional videos, results, and visualizations are available at https://hypernerf.github.io.

SIGGRAPH Asia 2021, Project page: https://hypernerf.github.io/

References in corpus (4)

• Auto-Encoding Variational Bayes
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Neural Non-Rigid Tracking
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video

Cited by in corpus (19)

• Fast Dynamic Radiance Fields with Time-Aware Neural Voxels
• HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
• NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads
• Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
• LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar
• Fluid Simulation on Neural Flow Maps
• EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes
• Artemis: Articulated Neural Pets with Appearance and Motion synthesis
• BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis
• GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
• CTNeRF: Cross-Time Transformer for Dynamic Neural Radiance Field from Monocular Video
• MARF: The Medial Atom Ray Field Object Representation
• EasyVolcap: Accelerating Neural Volumetric Video Research
• DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering
• Neural Fields in Visual Computing and Beyond
• Deblur-NeRF: Neural Radiance Fields from Blurry Images
• Animating NeRFs from Texture Space: A Framework for Pose-Dependent Rendering of Human Performances
• Learning to Stylize Novel Views
• VaxNeRF: Revisiting the Classic for Voxel-Accelerated Neural Radiance Field