Modeling Clothing as a Separate Layer for an Animatable Human Avatar

Abstract

We have recently seen great progress in building photorealistic animatable full-body codec avatars, but generating high-fidelity animation of clothing is still difficult. To address these difficulties, we propose a method to build an animatable clothed body avatar with an explicit representation of the clothing on the upper body from multi-view captured videos. We use a two-layer mesh representation to register each 3D scan separately with the body and clothing templates. In order to improve the photometric correspondence across different frames, texture alignment is then performed through inverse rendering of the clothing geometry and texture predicted by a variational autoencoder. We then train a new two-layer codec avatar with separate modeling of the upper clothing and the inner body layer. To learn the interaction between the body dynamics and clothing states, we use a temporal convolution network to predict the clothing latent code based on a sequence of input skeletal poses. We show photorealistic animation output for three different actors, and demonstrate the advantage of our clothed-body avatars over the single-layer avatars used in previous work. We also show the benefit of an explicit clothing model that allows the clothing texture to be edited in the animation output.

arXiv commentCamera ready for SIGGRAPH Asia 2021 Technical Papers. https://research.fb.com/publications/modeling-clothing-as-a-separate-layer-for-an-animatable-human-avatar/

References in corpus (5)

• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Deep Appearance Models for Face Rendering
• Real-time Deep Dynamic Characters
• Driving-Signal Aware Full-Body Avatars
• PBNS: Physically Based Neural Simulator for Unsupervised Garment Pose Space Deformation

Cited by in corpus (10)

• Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing
• PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling
• AvatarFusion: Zero-shot Generation of Clothing-Decoupled 3D Avatars Using 2D Diffusion
• Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input
• CaPhy: Capturing Physical Properties for Animatable Human Avatars
• Interactive Rendering of Relightable and Animatable Gaussian Avatars
• GGAvatar: Reconstructing Garment-Separated 3D Gaussian Splatting Avatars from Monocular Video
• Real-Time Per-Garment Virtual Try-On with Temporal Consistency for Loose-Fitting Garments
• Relightable and Dynamic Gaussian Avatar Reconstruction from Monocular Video
• Dress-Me-Up: A Dataset & Method for Self-Supervised 3D Garment Retargeting