HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion

Abstract

Representing human performance at high-fidelity is an essential building block in diverse applications, such as film production, computer games or videoconferencing. To close the gap to production-level quality, we introduce HumanRF, a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints. Our novel representation acts as a dynamic video encoding that captures fine details at high compression rates by factorizing space-time into a temporal matrix-vector decomposition. This allows us to obtain temporally coherent reconstructions of human actors for long sequences, while representing high-resolution details even in the context of challenging motion. While most research focuses on synthesizing at resolutions of 4MP or lower, we address the challenge of operating at 12MP. To this end, we introduce ActorsHQ, a novel multi-view dataset that provides 12MP footage from 160 cameras for 16 sequences with high-fidelity, per-frame mesh reconstructions. We demonstrate challenges that emerge from using such high-resolution data and show that our newly introduced HumanRF effectively leverages this data, making a significant step towards production-level quality novel view synthesis.

Project webpage: https://synthesiaresearch.github.io/humanrf Dataset webpage: https://www.actors-hq.com/ Video: https://www.youtube.com/watch?v=OTnhiLLE7io Code: https://github.com/synthesiaresearch/humanrf

References in corpus (13)

• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
• Fast Dynamic Radiance Fields with Time-Aware Neural Voxels
• Real-time Deep Dynamic Characters
• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
• DeVRF: Fast Deformable Voxel Radiance Fields for Dynamic Scenes
• Compressible-composable NeRF via Rank-residual Decomposition
• Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
• Artemis: Articulated Neural Pets with Appearance and Motion synthesis
• NerfAcc: A General NeRF Acceleration Toolbox
• A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
• Robust High-Resolution Video Matting with Temporal Guidance
• NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing

Cited by in corpus (14)

• 3D Gaussian as a New Era: A Survey
• NPGA: Neural Parametric Gaussian Avatars
• GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
• Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input
• HPC: Hierarchical Progressive Coding Framework for Volumetric Video
• DiVa-360: The Dynamic Visual Dataset for Immersive Neural Fields
• FPO++: Efficient Encoding and Rendering of Dynamic Neural Radiance Fields by Analyzing and Enhancing Fourier PlenOctrees
• InfiniHuman: Infinite 3D Human Creation with Precise Control
• HGC-Avatar: Hierarchical Gaussian Compression for Streamable Dynamic 3D Avatars
• A Fast Volumetric Capture and Reconstruction Pipeline for Dynamic Point Clouds and Gaussian Splats
• Scalable and High-Quality Neural Implicit Representation for 3D Reconstruction
• HumanRAM: Feed-forward Human Reconstruction and Animation Model using Transformers
• DeMapGS: Simultaneous Mesh Deformation and Surface Attribute Mapping via Gaussian Splatting
• Detail Enhanced Gaussian Splatting for Large-Scale Volumetric Capture