9 papers
STEER: Steerable Dyadic Head Avatars
Kartik Teotia, Helge Rhodin, Hyeongwoo Kim +2
Facial movement and expression are central to face-to-face communication, conveying turn-taking, attention, agreement, and engagement alongside speech. While speech-driven facial a…
Polyglot: Multilingual Style Preserving Speech-Driven Facial Animation
Federico Nocentini, Kwanggyoon Seo, Qingju Liu +6
Speech-Driven Facial Animation (SDFA) has gained significant attention due to its applications in movies, video games, and virtual reality. However, most existing models are traine…
Zero-Shot Video Deraining with Video Diffusion Models
Tuomas Varanka, Juan Luis Gonzalez, Hyeongwoo Kim +2
Existing video deraining methods are often trained on paired datasets, either synthetic, which limits their ability to generalize to real-world rain, or captured by static cameras,…
Audio-Driven Universal Gaussian Head Avatars
Kartik Teotia, Helge Rhodin, Mohit Mendiratta +3
We introduce the first method for audio-driven universal photorealistic avatar synthesis, combining a person-agnostic speech model with our novel Universal Head Avatar Prior (UHAP)…
Contextual Gesture: Co-Speech Gesture Video Generation through Context-aware Gesture Representation
Pinxin Liu, Pengfei Zhang, Hyeongwoo Kim +3
Co-speech gesture generation is crucial for creating lifelike avatars and enhancing human-computer interactions by synchronizing gestures with speech. Despite recent advancements,…
ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs
Eray Eren, Qingju Liu, Hyeongwoo Kim +2
Prosody conveys rich emotional and semantic information of the speech signal as well as individual idiosyncrasies. We propose a stand-alone model that maps text-to-prosodic feature…