1 citations · 2 across the 10 of their papers we have counts for
10 papers
S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modeling
Xueqi Wang, Zhigang Wang, Runqing Zhang +2
Multimodal dialogue retrieval aims to retrieve dialogues from multimodal dialogue banks that are similar to a target dialogue in terms of both textual semantics and acoustic conver…
AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis
Zhenqi Jia, Yuan Zhao, Aruukhan +2
Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression and contextual consistency in user-agent interactions. Existing CSS methods str…
MAR3: Multi-Agent Recognition, Reasoning, and Reflection for Reference Audio-Visual Segmentation
Yuan Zhao, Zhenqi Jia, Yongqiang Zhang
Reference Audio-Visual Segmentation (Ref-AVS) aims to segment objects in audible videos based on multimodal cues in reference expressions. Previous methods overlook the explicit re…
Emotion and Acoustics Should Agree: Cross-Level Inconsistency Analysis for Audio Deepfake Detection
Jinhua Zhang, Zhenqi Jia, Rui Liu
Audio Deepfake Detection (ADD) aims to detect spoof speech from bonafide speech. Most prior studies assume that stronger correlations within or across acoustic and emotional featur…
Towards Authentic Movie Dubbing with Retrieve-Augmented Director-Actor Interaction Learning
Rui Liu, Yuan Zhao, Zhenqi Jia
The automatic movie dubbing model generates vivid speech from given scripts, replicating a speaker's timbre from a brief timbre prompt while ensuring lip-sync with the silent video…
Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis
Zhenqi Jia, Rui Liu, Berrak Sisman +1
Conversational Speech Synthesis (CSS) aims to generate speech with natural prosody by understanding the multimodal dialogue history (MDH). The latest work predicts the accurate pro…
Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis
Rui Liu, Zhenqi Jia, Feilong Bao +1
Conversational speech synthesis (CSS) aims to take the current dialogue (CD) history as a reference to synthesize expressive speech that aligns with the conversational style. Unlik…
Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesis
Zhenqi Jia, Rui Liu
Conversational Speech Synthesis (CSS) aims to effectively take the multimodal dialogue history (MDH) to generate speech with appropriate conversational prosody for target utterance…
Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling
Rui Liu, Zhenqi Jia, Jie Yang +2
Conversational Text-to-Speech (CTTS) aims to accurately express an utterance with the appropriate style within a conversational setting, which attracts more attention nowadays. Whi…
MCDubber: Multimodal Context-Aware Expressive Video Dubbing
Yuan Zhao, Zhenqi Jia, Rui Liu +3
Automatic Video Dubbing (AVD) aims to take the given script and generate speech that aligns with lip motion and prosody expressiveness. Current AVD models mainly utilize visual inf…