DeepCap: Monocular Human Performance Capture Using Weak Supervision

Abstract

Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality. Many previous performance capture approaches either required expensive multi-view setups or did not recover dense space-time coherent geometry with frame-to-frame correspondences. We propose a novel deep learning approach for monocular dense human performance capture. Our method is trained in a weakly supervised manner based on multi-view supervision completely removing the need for training data with 3D ground truth annotations. The network architecture is based on two separate networks that disentangle the task into a pose estimation and a non-rigid surface deformation step. Extensive qualitative and quantitative evaluations show that our approach outperforms the state of the art in terms of quality and robustness.

References in corpus (10)

• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• Towards Accurate Markerless Human Shape and Pose Estimation over Time
• End-to-end Recovery of Human Shape and Pose
• Video Based Reconstruction of 3D People Models
• Monocular Total Capture: Posing Face, Body, and Hands in the Wild
• General Automatic Human Shape and Motion Capture Using Volumetric Contour Cues
• DeepHuman: 3D Human Reconstruction from a Single Image
• SiCloPe: Silhouette-Based Clothed People

Cited by in corpus (9)

• A Survey of Human-in-the-loop for Machine Learning
• A Survey on Deep Learning for Skeleton-Based Human Animation
• Shape Controllable Virtual Try-on for Underwear Models
• Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors
• Self-Supervised Collision Handling via Generative 3D Garment Models for Virtual Try-On
• Neural 3D Clothes Retargeting from a Single Image
• MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
• ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References
• Birds of a Feather: Capturing Avian Shape Models from Images