Sim2real transfer learning for 3D human pose estimation: motion to the rescue

Abstract

Synthetic visual data can provide practically infinite diversity and rich labels, while avoiding ethical issues with privacy and bias. However, for many tasks, current models trained on synthetic data generalize poorly to real data. The task of 3D human pose estimation is a particularly interesting example of this sim2real problem, because learning-based approaches perform reasonably well given real training data, yet labeled 3D poses are extremely difficult to obtain in the wild, limiting scalability. In this paper, we show that standard neural-network approaches, which perform poorly when trained on synthetic RGB images, can perform well when the data is pre-processed to extract cues about the person's motion, notably as optical flow and the motion of 2D keypoints. Therefore, our results suggest that motion can be a simple way to bridge a sim2real gap when video is available. We evaluate on the 3D Poses in the Wild dataset, the most challenging modern benchmark for 3D pose estimation, where we show full 3D mesh recovery that is on par with state-of-the-art methods trained on real 3D sequences, despite training only on synthetic humans from the SURREAL dataset.

arXiv commentAccepted at NeurIPS 2019

Cited by in corpus (21)

• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Deep Learning-Based Human Pose Estimation: A Survey
• Understanding Domain Randomization for Sim-to-real Transfer
• Inference Stage Optimization for Cross-scenario 3D Human Pose Estimation
• Playing for 3D Human Recovery
• DOVE: Learning Deformable 3D Objects by Watching Videos
• Domain Adaptation Through Task Distillation
• PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop
• Graph and Temporal Convolutional Networks for 3D Multi-person Pose Estimation in Monocular Videos
• Self-supervised Video Object Segmentation by Motion Grouping
• 3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning
• Bilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction
• 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos
• Self-supervised Tumor Segmentation through Layer Decomposition
• Fast Uncertainty Quantification for Deep Object Pose Estimation
• Monocular 3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks
• Image-to-Image Translation of Synthetic Samples for Rare Classes
• It's About Time: Analog Clock Reading in the Wild
• Learning Local Recurrent Models for Human Mesh Recovery
• Self-Attentive 3D Human Pose and Shape Estimation from Videos
• Synthetic Data Are as Good as the Real for Association Knowledge Learning in Multi-object Tracking