Synthesizing Training Images for Boosting Human 3D Pose Estimation

Abstract

Human 3D pose estimation from a single image is a challenging task with numerous applications. Convolutional Neural Networks (CNNs) have recently achieved superior performance on the task of 2D pose estimation from a single image, by training on images with 2D annotations collected by crowd sourcing. This suggests that similar success could be achieved for direct estimation of 3D poses. However, 3D poses are much harder to annotate, and the lack of suitable annotated training images hinders attempts towards end-to-end solutions. To address this issue, we opt to automatically synthesize training images with ground truth pose annotations. Our work is a systematic study along this road. We find that pose space coverage and texture diversity are the key ingredients for the effectiveness of synthetic training data. We present a fully automatic, scalable approach that samples the human pose space for guiding the synthesis procedure and extracts clothing textures from real images. Furthermore, we explore domain adaptation for bridging the gap between our synthetic training images and real testing photos. We demonstrate that CNNs trained with our synthetic images out-perform those trained with real photos on 3D pose estimation tasks.

References in corpus (10)

• Learning Transferable Features with Deep Adaptation Networks
• Unsupervised Domain Adaptation by Backpropagation
• Deep Domain Confusion: Maximizing for Domain Invariance
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations
• Convolutional Pose Machines
• R-CNNs for Pose Estimation and Action Detection
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• Combining Local Appearance and Holistic View: Dual-Source Deep Neural Networks for Human Pose Estimation
• Heterogeneous Multi-task Learning for Human Pose Estimation with Deep Convolutional Neural Network

Cited by in corpus (26)

• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Learning from Synthetic Humans
• FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
• Algorithms for Semantic Segmentation of Multispectral Remote Sensing Imagery using Deep Learning
• Self-supervised Learning of Motion Capture
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• BodyNet: Volumetric Inference of 3D Human Body Shapes
• Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera
• Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation
• Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision
• Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose
• Toward Characteristic-Preserving Image-based Virtual Try-On Network
• FBI-Pose: Towards Bridging the Gap between 2D Images and 3D Human Poses using Forward-or-Backward Information
• Towards Robust RGB-D Human Mesh Recovery
• Domain Adaptation from Synthesis to Reality in Single-model Detector for Video Smoke Detection
• Steady-state Non-Line-of-Sight Imaging
• SampleAhead: Online Classifier-Sampler Communication for Learning from Synthesized Data
• Toward Accurate and Realistic Virtual Try-on Through Shape Matching and Multiple Warps
• Ordinal Depth Supervision for 3D Human Pose Estimation
• Distill Knowledge from NRSfM for Weakly Supervised 3D Pose Learning
• Latent Space Representation for Shape Analysis and Learning
• Automated X-ray Image Analysis for Cargo Security: Critical Review and Future Promise
• Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB
• PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections
• Toward Accurate and Realistic Outfits Visualization with Attention to Details
• 3D Human Body Reconstruction from a Single Image via Volumetric Regression