5 papers · 1 filter
VSpeechLM: A Visual Speech Language Model for Visual Text-to-Speech Task
Yuyue Wang, Xin Cheng, Yihan Wu +3
The task of Visual Text-to-Speech (VisualTTS), also known as video dubbing, aims to generate speech synchronized with the lip movements in an input video, in additional to being co…
Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction
Kaisi Guan, Xihua Wang, Zhengfeng Lai +5
This study focuses on a challenging yet promising task, Text-to-Sounding-Video (T2SV) generation, which aims to generate a video with synchronized audio from text conditions, meanw…
VSSFlow: Unifying Video-conditioned Sound and Speech Generation via Joint Learning
Xin Cheng, Yuyue Wang, Xihua Wang +7
Video-conditioned audio generation, including Video-to-Sound (V2S) and Visual Text-to-Speech (VisualTTS), has traditionally been treated as distinct tasks, leaving the potential fo…
WildSpoof Challenge Evaluation Plan
Yihan Wu, Jee-weon Jung, Hye-jin Shim +2
The WildSpoof Challenge aims to advance the use of in-the-wild data in two intertwined speech processing tasks. It consists of two parallel tracks: (1) Text-to-Speech (TTS) synthes…
EyEar: Learning Audio Synchronized Human Gaze Trajectory Based on Physics-Informed Dynamics
Xiaochuan Liu, Xin Cheng, Yuchong Sun +4
Imitating how humans move their gaze in a visual scene is a vital research problem for both visual understanding and psychology, kindling crucial applications such as building aliv…