Deep Appearance Models for Face Rendering

Abstract

We introduce a deep appearance model for rendering the human face. Inspired by Active Appearance Models, we develop a data-driven rendering pipeline that learns a joint representation of facial geometry and appearance from a multiview capture setup. Vertex positions and view-specific textures are modeled using a deep variational autoencoder that captures complex nonlinear effects while producing a smooth and compact latent representation. View-specific texture enables the modeling of view-dependent effects such as specularity. In addition, it can also correct for imperfect geometry stemming from biased or low resolution estimates. This is a significant departure from the traditional graphics pipeline, which requires highly accurate geometry as well as all elements of the shading model to achieve realism through physically-inspired light transport. Acquiring such a high level of accuracy is difficult in practice, especially for complex and intricate parts of the face, such as eyelashes and the oral cavity. These are handled naturally by our approach, which does not rely on precise estimates of geometry. Instead, the shading model accommodates deficiencies in geometry though the flexibility afforded by the neural network employed. At inference time, we condition the decoding network on the viewpoint of the camera in order to generate the appropriate texture for rendering. The resulting system can be implemented simply using existing rendering engines through dynamic textures with flat lighting. This representation, together with a novel unsupervised technique for mapping images to facial states, results in a system that is naturally suited to real-time interactive settings such as Virtual Reality (VR).

arXiv commentAccepted to SIGGRAPH 2018

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Deep Convolutional Inverse Graphics Network
• Unsupervised Image-to-Image Translation Networks
• Variational Lossy Autoencoder
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data

Cited by in corpus (55)

• Neural Volumes: Learning Dynamic Renderable Volumes from Images
• GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction
• Real-time Deep Dynamic Characters
• Neural Head Reenactment with Latent Pose Descriptors
• A privacy-preserving approach to streaming eye-tracking data
• An Implicit Parametric Morphable Dental Model
• Neural Light Transport for Relighting and View Synthesis
• H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion
• Driving-Signal Aware Full-Body Avatars
• Modeling Clothing as a Separate Layer for an Animatable Human Avatar
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction
• Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
• The Security-Utility Trade-off for Iris Authentication and Eye Animation for Social Virtual Avatars
• PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization
• LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar
• Optimization and Manipulation of Contextual Mutual Spaces for Multi-User Virtual and Augmented Reality Interaction
• High-resolution Deep Convolutional Generative Adversarial Networks
• PVA: Pixel-aligned Volumetric Avatars
• A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
• FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
• Textured Neural Avatars
• Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input
• Going beyond Free Viewpoint: Creating Animatable Volumetric Video of Human Performances
• ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE
• Deep Deformable 3D Caricatures with Learned Shape Control
• VR Facial Animation for Immersive Telepresence Avatars
• Neural Parameterization for Dynamic Human Head Editing
• Deep Appearance Prefiltering
• Learning Compositional Radiance Fields of Dynamic Human Heads
• Efficient Large-Scale Face Clustering Using an Online Mixture of Gaussians
• ARCH++: Animation-Ready Clothed Human Reconstruction Revisited
• Light Stage Super-Resolution: Continuous High-Frequency Relighting
• EgoFace: Egocentric Face Performance Capture and Videorealistic Reenactment
• Robust Pose Transfer with Dynamic Details using Neural Video Rendering
• Expressive Telepresence via Modular Codec Avatars
• HUMBI: A Large Multiview Dataset of Human Body Expressions
• Fine Structure-Aware Sampling: A New Sampling Training Scheme for Pixel-Aligned Implicit Models in Single-View Human Reconstruction
• See360: Novel Panoramic View Interpolation
• Universal Facial Encoding of Codec Avatars from VR Headsets
• Towards Disentangled Representations for Human Retargeting by Multi-view Learning
• F-CAD: A Framework to Explore Hardware Accelerators for Codec Avatar Decoding
• Personalized Face Modeling for Improved Face Reconstruction and Motion Retargeting
• Learning Neural Light Transport
• Riggable 3D Face Reconstruction via In-Network Optimization
• REFA: Real-time Egocentric Facial Animations for Virtual Reality
• FVA: Modeling Perceived Friendliness of Virtual Agents Using Movement Characteristics
• Dual Grid Net: hand mesh vertex regression from single depth maps
• ViDA-MAN: Visual Dialog with Digital Humans
• One-Shot Domain Adaptation For Face Generation
• Self-Supervised Adaptation of High-Fidelity Face Models for Monocular Performance Tracking
• High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation
• EgoRenderer: Rendering Human Avatars from Egocentric Camera Images
• GSHeadRelight: Fast Relightability for 3D Gaussian Head Synthesis
• Dynamic Facial Asset and Rig Generation from a Single Scan
• An Image-based Generator Architecture for Synthetic Image Refinement