Robust Pose Transfer with Dynamic Details using Neural Video Rendering

Abstract

Pose transfer of human videos aims to generate a high fidelity video of a target person imitating actions of a source person. A few studies have made great progress either through image translation with deep latent features or neural rendering with explicit 3D features. However, both of them rely on large amounts of training data to generate realistic results, and the performance degrades on more accessible internet videos due to insufficient training frames. In this paper, we demonstrate that the dynamic details can be preserved even trained from short monocular videos. Overall, we propose a neural video rendering framework coupled with an image-translation-based dynamic details generation network (D2G-Net), which fully utilizes both the stability of explicit 3D features and the capacity of learning components. To be specific, a novel texture representation is presented to encode both the static and pose-varying appearance characteristics, which is then mapped to the image space and rendered as a detail-rich frame in the neural rendering stage. Moreover, we introduce a concise temporal loss in the training stage to suppress the detail flickering that is made more visible due to high-quality dynamic details generated by our method. Through extensive comparisons, we demonstrate that our neural human video renderer is capable of achieving both clearer dynamic details and more robust performance even on accessible short videos with only 2k - 4k frames.

arXiv commentVideo link: https://www.bilibili.com/video/BV1y64y1C7ge/

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Pose Guided Person Image Generation
• High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
• Deep Appearance Models for Face Rendering
• First Order Motion Model for Image Animation
• Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation
• Human Motion Transfer with 3D Constraints and Detail Enhancement
• Textured Neural Avatars

Cited by in corpus (1)

• Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis