GestureGAN for Hand Gesture-to-Gesture Translation in the Wild
arXiv:1808.04859
Abstract
Hand gesture-to-gesture translation in the wild is a challenging task since hand gestures can have arbitrary poses, sizes, locations and self-occlusions. Therefore, this task requires a high-level understanding of the mapping between the input source gesture and the output target gesture. To tackle this problem, we propose a novel hand Gesture Generative Adversarial Network (GestureGAN). GestureGAN consists of a single generator and a discriminator , which takes as input a conditional hand image and a target hand skeleton image. GestureGAN utilizes the hand skeleton information explicitly, and learns the gesture-to-gesture mapping through two novel losses, the color loss and the cycle-consistency loss. The proposed color loss handles the issue of "channel pollution" while back-propagating the gradients. In addition, we present the Fréchet ResNet Distance (FRD) to evaluate the quality of generated images. Extensive experiments on two widely used benchmark datasets demonstrate that the proposed GestureGAN achieves state-of-the-art performance on the unconstrained hand gesture-to-gesture translation task. Meanwhile, the generated images are in high-quality and are photo-realistic, allowing them to be used as data augmentation to improve the performance of a hand gesture classifier. Our model and code are available at https://github.com/Ha0Tang/GestureGAN.
arXiv comment9 pages, 7 figures, accepted to ACM MM 2018 as an oral paper, fix typos
References in corpus (24)
- Conditional Generative Adversarial Nets
- WaveNet: A Generative Model for Raw Audio
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Generative Adversarial Text to Image Synthesis
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- Improved Techniques for Training GANs
- Toward Multimodal Image-to-Image Translation
- BEGAN: Boundary Equilibrium Generative Adversarial Networks
- Pose Guided Person Image Generation
- Invertible Conditional GANs for image editing
- Semantic Segmentation using Adversarial Networks
- DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
- Dual Discriminator Generative Adversarial Nets
- Perceptual Generative Adversarial Networks for Small Object Detection
- GP-GAN: Towards Realistic High-Resolution Image Blending
- Controllable Invariance through Adversarial Feature Learning
- Geometry-Contrastive GAN for Facial Expression Transfer
- Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
- Neural Face Editing with Intrinsic Image Disentangling
- Face Transfer with Generative Adversarial Network
- Packing and Padding: Coupled Multi-index for Accurate Image Retrieval
- Every Smile is Unique: Landmark-Guided Diverse Smile Generation
- Geometry Guided Adversarial Facial Expression Synthesis
- Cross-domain Human Parsing via Adversarial Feature and Label Adaptation