MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
arXiv:2009.10711
Abstract
We present a method to capture temporally coherent dynamic clothing deformation from a monocular RGB video input. In contrast to the existing literature, our method does not require a pre-scanned personalized mesh template, and thus can be applied to in-the-wild videos. To constrain the output to a valid deformation space, we build statistical deformation models for three types of clothing: T-shirt, short pants and long pants. A differentiable renderer is utilized to align our captured shapes to the input frames by minimizing the difference in both silhouette, segmentation, and texture. We develop a UV texture growing method which expands the visible texture region of the clothing sequentially in order to minimize drift in deformation tracking. We also extract fine-grained wrinkle detail from the input videos by fitting the clothed surface to the normal maps estimated by a convolutional neural network. Our method produces temporally coherent reconstruction of body and clothing from monocular video. We demonstrate successful clothing capture results from a variety of challenging videos. Extensive quantitative experiments demonstrate the effectiveness of our method on metrics including body pose error and surface reconstruction error of the clothing.
arXiv comment3DV 2020 Camera Ready
References in corpus (10)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
- Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning
- Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research
- OriNet: A Fully Convolutional Network for 3D Human Pose Estimation
- Monocular Total Capture: Posing Face, Body, and Hands in the Wild
- Detailed, accurate, human shape estimation from clothed 3D scan sequences
- DeepCap: Monocular Human Performance Capture Using Weak Supervision