Pose Guided Human Video Generation

Abstract

Due to the emergence of Generative Adversarial Networks, video synthesis has witnessed exceptional breakthroughs. However, existing methods lack a proper representation to explicitly control the dynamics in videos. Human pose, on the other hand, can represent motion patterns intrinsically and interpretably, and impose the geometric constraints regardless of appearance. In this paper, we propose a pose guided method to synthesize human videos in a disentangled way: plausible motion prediction and coherent appearance generation. In the first stage, a Pose Sequence Generative Adversarial Network (PSGAN) learns in an adversarial manner to yield pose sequences conditioned on the class label. In the second stage, a Semantic Consistent Generative Adversarial Network (SCGAN) generates video frames from the poses while preserving coherent appearances in the input image. By enforcing semantic consistency between the generated and ground-truth poses at a high feature level, our SCGAN is robust to noisy or abnormal poses. Extensive experiments on both human action and human face datasets manifest the superiority of the proposed method over other state-of-the-arts.

arXiv commentAccepted to ECCV 2018

References in corpus (12)

• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Unsupervised Learning of Video Representations using LSTMs
• Generating Videos with Scene Dynamics
• Action-Conditional Video Prediction using Deep Networks in Atari Games
• Unsupervised Learning of Disentangled Representations from Video
• Recurrent Network Models for Human Dynamics
• Temporal Generative Adversarial Nets with Singular Value Clipping

Cited by in corpus (6)

• Video-to-Video Synthesis
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Vid2Game: Controllable Characters Extracted from Real-World Videos
• Conditional Video Generation Using Action-Appearance Captions
• G3AN: Disentangling Appearance and Motion for Video Generation
• Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency