Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation

Abstract

Synthesizing realistic videos of humans using neural networks has been a popular alternative to the conventional graphics-based rendering pipeline due to its high efficiency. Existing works typically formulate this as an image-to-image translation problem in 2D screen space, which leads to artifacts such as over-smoothing, missing body parts, and temporal instability of fine-scale detail, such as pose-dependent wrinkles in the clothing. In this paper, we propose a novel human video synthesis method that approaches these limiting factors by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. More specifically, our method relies on the combination of two convolutional neural networks (CNNs). Given the pose information, the first CNN predicts a dynamic texture map that contains time-coherent high-frequency details, and the second CNN conditions the generation of the final video on the temporally coherent output of the first CNN. We demonstrate several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.

References in corpus (8)

• Conditional Generative Adversarial Nets
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Pose Guided Person Image Generation
• Deep Appearance Models for Face Rendering
• DeepWrinkles: Accurate and Realistic Clothing Modeling
• Textured Neural Avatars

Cited by in corpus (12)

• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
• Style and Pose Control for Image Synthesis of Humans from a Single Monocular View
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild
• Space-time Neural Irradiance Fields for Free-Viewpoint Video
• Neural Sparse Voxel Fields
• AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
• Robust Pose Transfer with Dynamic Details using Neural Video Rendering
• HumanGAN: A Generative Model of Humans Images
• Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies
• Efficient Neural Radiance Fields for Interactive Free-viewpoint Video
• EgoRenderer: Rendering Human Avatars from Egocentric Camera Images