PVA: Pixel-aligned Volumetric Avatars

Abstract

Acquisition and rendering of photo-realistic human heads is a highly challenging research problem of particular importance for virtual telepresence. Currently, the highest quality is achieved by volumetric approaches trained in a person specific manner on multi-view data. These models better represent fine structure, such as hair, compared to simpler mesh-based models. Volumetric models typically employ a global code to represent facial expressions, such that they can be driven by a small set of animation parameters. While such architectures achieve impressive rendering quality, they can not easily be extended to the multi-identity setting. In this paper, we devise a novel approach for predicting volumetric avatars of the human head given just a small number of inputs. We enable generalization across identities by a novel parameterization that combines neural radiance fields with local, pixel-aligned features extracted directly from the inputs, thus sidestepping the need for very deep or complex networks. Our approach is trained in an end-to-end manner solely based on a photometric re-rendering loss without requiring explicit 3D supervision.We demonstrate that our approach outperforms the existing state of the art in terms of quality and is able to generate faithful facial expressions in a multi-identity setting.

Project page located at https://volumetric-avatars.github.io/

References in corpus (7)

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Deep Appearance Models for Face Rendering
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• StyleGAN2 Distillation for Feed-forward Image Manipulation

Cited by in corpus (9)

• Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
• Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
• Style and Pose Control for Image Synthesis of Humans from a Single Monocular View
• DOVE: Learning Deformable 3D Objects by Watching Videos
• AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
• Neural Fields in Visual Computing and Beyond
• Neural Rays for Occlusion-aware Image-based Rendering
• Learning to Stylize Novel Views