FaceVR: Real-Time Facial Reenactment and Eye Gaze Control in Virtual Reality

Abstract

We propose FaceVR, a novel image-based method that enables video teleconferencing in VR based on self-reenactment. State-of-the-art face tracking methods in the VR context are focused on the animation of rigged 3d avatars. While they achieve good tracking performance the results look cartoonish and not real. In contrast to these model-based approaches, FaceVR enables VR teleconferencing using an image-based technique that results in nearly photo-realistic outputs. The key component of FaceVR is a robust algorithm to perform real-time facial motion capture of an actor who is wearing a head-mounted display (HMD), as well as a new data-driven approach for eye tracking from monocular videos. Based on reenactment of a prerecorded stereo video of the person without the HMD, FaceVR incorporates photo-realistic re-rendering in real time, thus allowing artificial modifications of face and eye appearances. For instance, we can alter facial expressions or change gaze directions in the prerecorded target video. In a live setup, we apply these newly-introduced algorithmic components.

Video: https://youtu.be/jIlujM5avU8 Presented at Siggraph'18

References in corpus (3)

• Automatic Face Reenactment
• Real-Time Facial Segmentation and Performance Capture from RGB Input
• Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging

Cited by in corpus (12)

• A Review and Analysis of Eye-Gaze Estimation Systems, Algorithms and Performance Evaluation Methods in Consumer Platforms
• On Learning 3D Face Morphable Model from In-the-wild Images
• EyeNeRF: A Hybrid Representation for Photorealistic Synthesis, Animation and Relighting of Human Eyes
• ReenactGAN: Learning to Reenact Faces via Boundary Transfer
• Identity Preserving Face Completion for Large Ocular Region Occlusion
• Unconstrained Facial Expression Transfer using Style-based Generator
• Mask-off: Synthesizing Face Images in the Presence of Head-mounted Displays
• Expression-aware video inpainting for HMD removal in XR applications
• Frontal Screens on Head-Mounted Displays to Increase Awareness of the HMD Users' State in Mixed Presence Collaboration
• Die Zukunft sehen: Die Chancen und Herausforderungen der Erweiterten und Virtuellen Realität für industrielle Anwendungen
• IMU2Face: Real-time Gesture-driven Facial Reenactment
• EgoRenderer: Rendering Human Avatars from Egocentric Camera Images