Self-supervised Learning of Motion Capture

Abstract

Current state-of-the-art solutions for motion capture from a single camera are optimization driven: they optimize the parameters of a 3D human model so that its re-projection matches measurements in the video (e.g. person segmentation, optical flow, keypoint detections etc.). Optimization models are susceptible to local minima. This has been the bottleneck that forced using clean green-screen like backgrounds at capture time, manual initialization, or switching to multiple cameras as input resource. In this work, we propose a learning based motion capture model for single camera input. Instead of optimizing mesh and skeleton parameters directly, our model optimizes neural network weights that predict 3D shape and skeleton configurations given a monocular RGB video. Our model is trained using a combination of strong supervision from synthetic data, and self-supervision from differentiable rendering of (a) skeletal keypoints, (b) dense 3D mesh motion, and (c) human-background segmentation, in an end-to-end framework. Empirically we show our model combines the best of both worlds of supervised learning and test-time optimization: supervised learning initializes the model parameters in the right regime, ensuring good pose and surface initialization at test time, without manual effort. Self-supervision by back-propagating through differentiable rendering allows (unsupervised) adaptation of the model to the test data, and offers much tighter fit than a pretrained fixed model. We show that the proposed model improves with experience and converges to low-error solutions where previous optimization methods fail.

Neural Information Processing Systems (NIPS) 2017

References in corpus (12)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Learning from Synthetic Humans
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• SfM-Net: Learning of Structure and Motion from Video
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Convolutional Pose Machines
• Spatio-temporal video autoencoder with differentiable memory
• Unsupervised Learning of Depth and Ego-Motion from Video
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Synthesizing Training Images for Boosting Human 3D Pose Estimation
• MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior
• 3D Human Pose Estimation = 2D Pose Estimation + Matching

Cited by in corpus (6)

• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
• Active learning for interactive satellite image change detection
• Personalized 3D Human Pose and Shape Refinement
• 3D Human Mesh Regression with Dense Correspondence
• Monocular Human Shape and Pose with Dense Mesh-borne Local Image Features