MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild

Abstract

This paper addresses the problem of 3D human pose estimation in the wild. A significant challenge is the lack of training data, i.e., 2D images of humans annotated with 3D poses. Such data is necessary to train state-of-the-art CNN architectures. Here, we propose a solution to generate a large set of photorealistic synthetic images of humans with 3D pose annotations. We introduce an image-based synthesis engine that artificially augments a dataset of real images with 2D human pose annotations using 3D Motion Capture (MoCap) data. Given a candidate 3D pose our algorithm selects for each joint an image whose 2D pose locally matches the projected 3D pose. The selected images are then combined to generate a new synthetic image by stitching local image patches in a kinematically constrained manner. The resulting images are used to train an end-to-end CNN for full-body 3D pose estimation. We cluster the training data into a large number of pose classes and tackle pose estimation as a K-way classification problem. Such an approach is viable only with large training sets such as ours. Our method outperforms the state of the art in terms of 3D pose estimation in controlled environments (Human3.6M) and shows promising results for in-the-wild images (LSP). This demonstrates that CNNs trained on artificial images generalize well to real images.

9 pages, accepted to appear in NIPS 2016

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Convolutional Pose Machines
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation

Cited by in corpus (47)

• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Learning from Synthetic Humans
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
• Self-supervised Learning of Motion Capture
• AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild
• A survey of synthetic data augmentation methods in computer vision
• A simple yet effective baseline for 3d human pose estimation
• Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars
• SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• BodyNet: Volumetric Inference of 3D Human Body Shapes
• OriNet: A Fully Convolutional Network for 3D Human Pose Estimation
• Single-shot 3D multi-person pose estimation in complex images
• Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation
• Monocular Total Capture: Posing Face, Body, and Hands in the Wild
• Learning to Estimate 3D Human Pose and Shape from a Single Color Image
• Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera
• SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach
• Integral Human Pose Regression
• Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation
• Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision
• Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose
• 3D Human Pose Estimation from a Single Image via Distance Matrix Regression
• 3D Human Pose Estimation = 2D Pose Estimation + Matching
• PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation
• FBI-Pose: Towards Bridging the Gap between 2D Images and 3D Human Poses using Forward-or-Backward Information
• Structure from Articulated Motion: Accurate and Stable Monocular 3D Reconstruction without Training Data
• Towards Robust RGB-D Human Mesh Recovery
• DRPose3D: Depth Ranking in 3D Human Pose Estimation
• Ego-Pose Estimation and Forecasting as Real-Time PD Control
• Frankenstein: Learning Deep Face Representations using Small Data
• Human Image Generation: A Comprehensive Survey
• Learning 3D Human Pose from Structure and Motion
• Improving Face Anti-Spoofing by 3D Virtual Synthesis
• Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture
• Ordinal Depth Supervision for 3D Human Pose Estimation
• Predicting Animation Skeletons for 3D Articulated Models via Volumetric Nets
• EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera
• RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream
• Toward Marker-free 3D Pose Estimation in Lifting: A Deep Multi-view Solution
• Deep Pose Consensus Networks
• Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB
• Skeleton-Graph: Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs
• PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections
• VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space
• Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose