Disentangling Pose from Appearance in Monochrome Hand Images
arXiv:1904.07528
Abstract
Hand pose estimation from the monocular 2D image is challenging due to the variation in lighting, appearance, and background. While some success has been achieved using deep neural networks, they typically require collecting a large dataset that adequately samples all the axes of variation of hand images. It would, therefore, be useful to find a representation of hand pose which is independent of the image appearance~(like hand texture, lighting, background), so that we can synthesize unseen images by mixing pose-appearance combinations. In this paper, we present a novel technique that disentangles the representation of pose from a complementary appearance factor in 2D monochrome images. We supervise this disentanglement process using a network that learns to generate images of hand using specified pose+appearance features. Unlike previous work, we do not require image pairs with a matching pose; instead, we use the pose annotations already available and introduce a novel use of cycle consistency to ensure orthogonality between the factors. Experimental results show that our self-disentanglement scheme successfully decomposes the hand image into the pose and its complementary appearance features of comparable quality as the method using paired data. Additionally, training the model with extra synthesized images with unseen hand-appearance combinations by re-mixing pose and appearance factors from different images can improve the 2D pose estimation performance.
arXiv comment10 pages, 6 figures
References in corpus (14)
- Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections
- Improved Techniques for Training GANs
- Deep Image Prior
- Learning Deconvolution Network for Semantic Segmentation
- Stacked Hourglass Networks for Human Pose Estimation
- Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
- Deep Identity-aware Transfer of Facial Attributes
- Image-to-image translation for cross-domain disentanglement
- Challenges in Disentangling Independent Factors of Variation
- Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
- Visual Question Generation as Dual Task of Visual Question Answering
- Disentangled Person Image Generation
- Disentangling Latent Hands for Image Synthesis and Pose Estimation
- Disentangling Factors of Variation by Mixing Them