Deferred Neural Rendering: Image Synthesis using Neural Textures

Abstract

The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering, a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures, which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.

Video: https://youtu.be/z-pVip6WeyY SIGGRAPH 2019

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Conditional Image Generation with PixelCNN Decoders
• Deep Convolutional Inverse Graphics Network
• FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
• Transformation Properties of Learned Visual Representations
• Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
• IGNOR: Image-guided Neural Object Rendering

Cited by in corpus (15)

• FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping
• Deep Convolutional Pooling Transformer for Deepfake Detection
• A Review of Deep Learning-based Approaches for Deepfake Content Detection
• DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms
• DeepFake Detection Based on the Discrepancy Between the Face and its Context
• One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
• Equivariant Neural Rendering
• Face X-ray for More General Face Forgery Detection
• NASA: Neural Articulated Shape Approximation
• A Convolutional LSTM based Residual Network for Deepfake Video Detection
• AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection
• Adversarial Texture Optimization from RGB-D Scans
• One-Shot Identity-Preserving Portrait Reenactment
• A Neural Rendering Framework for Free-Viewpoint Relighting
• Sparse to Dense Motion Transfer for Face Image Animation