NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads

Abstract

We focus on reconstructing high-fidelity radiance fields of human heads, capturing their animations over time, and synthesizing re-renderings from novel viewpoints at arbitrary time steps. To this end, we propose a new multi-view capture setup composed of 16 calibrated machine vision cameras that record time-synchronized images at 7.1 MP resolution and 73 frames per second. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads, from which we introduce a new human head reconstruction benchmark. The recorded sequences cover a wide range of facial dynamics, including head motions, natural expressions, emotions, and spoken language. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles (NeRSemble). We represent scene dynamics by combining a deformation field and an ensemble of 3D multi-resolution hash encodings. The deformation field allows for precise modeling of simple scene movements, while the ensemble of hash encodings helps to represent complex dynamics. As a result, we obtain radiance field representations of human heads that capture motion over time and facilitate re-rendering of arbitrary novel viewpoints. In a series of experiments, we explore the design choices of our method and demonstrate that our approach outperforms state-of-the-art dynamic radiance field approaches by a significant margin.

Siggraph 2023, Project Page: https://tobias-kirschstein.github.io/nersemble/ , Video: https://youtu.be/a-OAWqBzldU

References in corpus (8)

• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
• Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video
• ReLU Fields: The Little Non-linearity That Could
• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
• Multiface: A Dataset for Neural Face Rendering
• Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
• Streaming Radiance Fields for 3D Video Synthesis
• NerfAcc: A General NeRF Acceleration Toolbox

Cited by in corpus (9)

• NPGA: Neural Parametric Gaussian Avatars
• Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos
• Cafca: High-quality Novel View Synthesis of Expressive Faces from Casual Few-shot Captures
• Learning a Generalized Physical Face Model From Data
• FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces
• Text-based Animatable 3D Avatars with Morphable Model Alignment
• Efficient Label Refinement for Face Parsing Under Extreme Poses Using 3D Gaussian Splatting
• Leveraging Avatar Fingerprinting: A Multi-Generator Photorealistic Talking-Head Public Database and Benchmark
• FFAvatar: Feed-Forward 4D Head Avatar Reconstruction from Sparse Portrait Images