Video Based Reconstruction of 3D People Models

Abstract

This paper describes how to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving. Based on a parametric body model, we present a robust processing pipeline achieving 3D model fits with 5mm accuracy also for clothed people. Our main contribution is a method to nonrigidly deform the silhouette cones corresponding to the dynamic human silhouettes, resulting in a visual hull in a common reference frame that enables surface reconstruction. This enables efficient estimation of a consensus 3D shape, texture and implanted animation skeleton based on a large number of frames. We present evaluation results for a number of test subjects and analyze overall performance. Requiring only a smartphone or webcam, our method enables everyone to create their own fully animatable digital double, e.g., for social VR applications or virtual try-on for online fashion shopping.

arXiv commentCVPR 2018 Spotlight, IEEE Conference on Computer Vision and Pattern Recognition 2018 (CVPR)

References in corpus (10)

• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Convolutional Pose Machines
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• Compositional Human Pose Regression
• Estimation of Human Body Shape and Posture Under Clothing
• Unite the People: Closing the Loop Between 3D and 2D Human Representations
• Detailed, accurate, human shape estimation from clothed 3D scan sequences
• General Automatic Human Shape and Motion Capture Using Volumetric Contour Cues
• Model-based Outdoor Performance Capture

Cited by in corpus (11)

• DeepCap: Monocular Human Performance Capture Using Weak Supervision
• Detailed Human Avatars from Monocular Video
• Recovering and Simulating Pedestrians in the Wild
• Photo Wake-Up: 3D Character Animation from a Single Photo
• Recovering Geometric Information with Learned Texture Perturbations
• 3D Human Texture Estimation from a Single Image with Transformers
• Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation
• Learning Parallel Dense Correspondence from Spatio-Temporal Descriptors for Efficient and Robust 4D Reconstruction
• Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB
• Fashion is Taking Shape: Understanding Clothing Preference Based on Body Shape From Online Sources
• A Shape-Aware Retargeting Approach to Transfer Human Motion and Appearance in Monocular Videos