Driving-Signal Aware Full-Body Avatars

Abstract

We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driving signals, such as human pose and facial keypoints, and produces a high-quality representation of human geometry and view-dependent appearance. The core intuition behind our method is that better drivability and generalization can be achieved by disentangling the driving signals and remaining generative factors, which are not available during animation. To this end, we explicitly account for information deficiency in the driving signal by introducing a latent space that exclusively captures the remaining information, thus enabling the imputation of the missing factors required during full-body animation, while remaining faithful to the driving signal. We also propose a learnable localized compression for the driving signal which promotes better generalization, and helps minimize the influence of global chance-correlations often found in real datasets. For a given driving signal, the resulting variational model produces a compact space of uncertainty for missing factors that allows for an imputation strategy best suited to a particular application. We demonstrate the efficacy of our approach on the challenging problem of full-body animation for virtual telepresence with driving signals acquired from minimal sensors placed in the environment and mounted on a VR-headset.

References in corpus (5)

• Geometric deep learning: going beyond Euclidean data
• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity
• Deep Appearance Models for Face Rendering
• Fader Networks: Manipulating Images by Sliding Attributes
• Disentangled Human Body Embedding Based on Deep Hierarchical Neural Network

Cited by in corpus (6)

• Modeling Clothing as a Separate Layer for an Animatable Human Avatar
• Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing
• PoseVocab: Learning Joint-structured Pose Embeddings for Human Avatar Modeling
• Single-Image 3D Human Digitization with Shape-Guided Diffusion
• Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input
• Subjective and Objective Quality Assessment of Rendered Human Avatar Videos in Virtual Reality