680 citations · 2.5k across the 34 of their papers we have counts for
39 papers
RelightableHands: Efficient Neural Relighting of Articulated Hand Models
Shun Iwase, Shunsuke Saito, Tomas Simon +7
We present the first neural relighting approach for rendering high-fidelity personalized hands that can be animated in real-time under novel illumination. Our approach adopts a tea…
Drivable Volumetric Avatars using Texel-Aligned Features
Edoardo Remelli, Timur Bagautdinov, Shunsuke Saito +8
Photorealistic telepresence requires both high-fidelity body modeling and faithful driving to enable dynamically synthesized appearance that is indistinguishable from reality. In t…
Multiface: A Dataset for Neural Face Rendering
Cheng-hsin Wuu, Ningyuan Zheng, Scott Ardisson +28
Photorealistic avatars of human faces have come a long way in recent years, yet research along this area is limited by a lack of publicly available, high-quality datasets covering…
Garment Avatars: Realistic Cloth Driving using Pattern Registration
Oshri Halimi, Fabian Prada, Tuur Stuyck +7
Virtual telepresence is the future of online communication. Clothing is an essential part of a person's identity and self-expression. Yet, ground truth data of registered clothes i…
Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing
Donglai Xiang, Timur Bagautdinov, Tuur Stuyck +10
Despite recent progress in developing animatable full-body avatars, realistic modeling of clothing - one of the core aspects of human self-expression - remains an open challenge. S…
Driving-Signal Aware Full-Body Avatars
Timur Bagautdinov, Chenglei Wu, Tomas Simon +6
We present a learning-based method for building driving-signal aware full-body avatars. Our model is a conditional variational autoencoder that can be animated with incomplete driv…
Pixel Codec Avatars
Shugao Ma, Tomas Simon, Jason Saragih +4
Telecommunication with photorealistic avatars in virtual or augmented reality is a promising path for achieving authentic face-to-face communication in 3D over remote physical dist…
MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement
Alexander Richard, Michael Zollhoefer, Yandong Wen +2
This paper presents a generic method for generating full facial 3D animation from speech. Existing approaches to audio-driven facial animation exhibit uncanny or static upper face…
High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation
Lele Chen, Chen Cao, Fernando De la Torre +3
3D video avatars can empower virtual communications by providing compression, privacy, entertainment, and a sense of presence in AR/VR. Best 3D photo-realistic AR/VR avatars driven…
Mixture of Volumetric Primitives for Efficient Neural Rendering
Stephen Lombardi, Tomas Simon, Gabriel Schwartz +3
Real-time rendering and animation of humans is a core function in games, movies, and telepresence applications. Existing methods have a number of drawbacks we aim to address with o…
Supervision by Registration and Triangulation for Landmark Detection
Xuanyi Dong, Yi Yang, Shih-En Wei +3
We present Supervision by Registration and Triangulation (SRT), an unsupervised approach that utilizes unlabeled multi-view video to improve the accuracy and precision of landmark…
Expressive Telepresence via Modular Codec Avatars
Hang Chu, Shugao Ma, Fernando De la Torre +2
VR telepresence consists of interacting with another human in a virtual space represented by an avatar. Today most avatars are cartoon-like, but soon the technology will allow vide…
Audio- and Gaze-driven Facial Animation of Codec Avatars
Alexander Richard, Colin Lea, Shugao Ma +3
Codec Avatars are a recent class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D (i.e., for virtual reality), and are al…
Spatiotemporal Bundle Adjustment for Dynamic 3D Human Reconstruction in the Wild
Minh Vo, Yaser Sheikh, Srinivasa G. Narasimhan
Bundle adjustment jointly optimizes camera intrinsics and extrinsics and 3D point triangulation to reconstruct a static scene. The triangulation constraint, however, is invalid for…
Fully Convolutional Mesh Autoencoder using Efficient Spatially Varying Kernels
Yi Zhou, Chenglei Wu, Zimo Li +5
Learning latent representations of registered meshes is useful for many 3D tasks. Techniques have recently shifted to neural mesh autoencoders. Although they demonstrate higher pre…
4D Visualization of Dynamic Events from Unconstrained Multi-View Videos
Aayush Bansal, Minh Vo, Yaser Sheikh +2
We present a data-driven approach for 4D space-time visualization of dynamic events from videos captured by hand-held multiple cameras. Key to our approach is the use of self-super…
To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations
Chaitanya Ahuja, Shugao Ma, Louis-Philippe Morency +1
Non verbal behaviours such as gestures, facial expressions, body posture, and para-linguistic cues have been shown to complement or clarify verbal messages. Hence to improve telepr…
Single-Network Whole-Body Pose Estimation
Gines Hidalgo, Yaadhav Raaj, Haroon Idrees +4
We present the first single-network approach for 2D~whole-body pose estimation, which entails simultaneous localization of body, face, hands, and feet keypoints. Due to the bottom-…
Neural Volumes: Learning Dynamic Renderable Volumes from Images
Stephen Lombardi, Tomas Simon, Jason Saragih +3
Modeling and rendering of dynamic scenes is challenging, as natural scenes often contain complex phenomena such as thin structures, evolving topology, translucency, scattering, occ…
Shapes and Context: In-the-Wild Image Synthesis & Manipulation
Aayush Bansal, Yaser Sheikh, Deva Ramanan
We introduce a data-driven approach for interactively synthesizing in-the-wild images from semantic label maps. Our approach is dramatically different from recent work in this spac…
Towards Social Artificial Intelligence: Nonverbal Social Signal Prediction in A Triadic Interaction
Hanbyul Joo, Tomas Simon, Mina Cikara +1
We present a new research task and a dataset to understand human social interactions via computational methods, to ultimately endow machines with the ability to encode and decode a…
LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
Chun-Liang Li, Tomas Simon, Jason Saragih +2
We present LBS-AE; a self-supervised autoencoding algorithm for fitting articulated mesh models to point clouds. As input, we take a sequence of point clouds to be registered as we…
Capture Dense: Markerless Motion Capture Meets Dense Pose Estimation
Xiu Li, Yebin Liu, Hanbyul Joo +2
We present a method to combine markerless motion capture and dense pose feature estimation into a single framework. We demonstrate that dense pose information can help for multivie…
Monocular Total Capture: Posing Face, Body, and Hands in the Wild
Donglai Xiang, Hanbyul Joo, Yaser Sheikh
We present the first method to capture the 3D total motion of a target person from a monocular view input. Given an image or a monocular video, our method reconstructs the motion f…