36 citations · 50 across the 9 of their papers we have counts for
13 papers
Follow the Clues, Frame the Truth: Hybrid-evidential Deductive Reasoning in Open-Vocabulary Multimodal Emotion Recognition
Yu Liu, Lei Zhang, Haoxun Li +4
Open-Vocabulary Multimodal Emotion Recognition (OV-MER) is inherently challenging due to the ambiguity of equivocal multimodal cues, which often stem from distinct unobserved situa…
Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations
Yu Liu, Hanlei Shi, Haoxun Li +5
Emotion Recognition in Conversations (ERC) is hard because discriminative evidence is sparse, localized, and often asynchronous across modalities. We center ERC on emotion hotspots…
EMORL-TTS: Reinforcement Learning for Fine-Grained Emotion Control in LLM-based TTS
Haoxun Li, Yu Liu, Yuqing Sun +3
Recent LLM-based TTS systems achieve strong quality and zero-shot ability, but lack fine-grained emotional control due to their reliance on discrete speech tokens. Existing approac…
MSF-SER: Enriching Acoustic Modeling with Multi-Granularity Semantics for Speech Emotion Recognition
Haoxun Li, Yuqing Sun, Hanlei Shi +3
Continuous dimensional speech emotion recognition captures affective variation along valence, arousal, and dominance, providing finer-grained representations than categorical appro…
Think-Before-Draw: Decomposing Emotion Semantics & Fine-Grained Controllable Expressive Talking Head Generation
Hanlei Shi, Leyuan Qu, Yu Liu +3
Emotional talking-head generation has emerged as a pivotal research area at the intersection of computer vision and multimodal artificial intelligence, with its core value lying in…
EME-TTS: Unlocking the Emphasis and Emotion Link in Speech Synthesis
Haoxun Li, Leyuan Qu, Jiaxi Hu +1
In recent years, emotional Text-to-Speech (TTS) synthesis and emphasis-controllable speech synthesis have advanced significantly. However, their interaction remains underexplored.…
From Coarse to Nuanced: Cross-Modal Alignment of Fine-Grained Linguistic Cues and Visual Salient Regions for Dynamic Emotion Recognition
Yu Liu, Leyuan Qu, Hanlei Shi +3
Dynamic Facial Expression Recognition (DFER) aims to identify human emotions from temporally evolving facial movements and plays a critical role in affective computing. While recen…
Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
Leyuan Qu, Cornelius Weber, Stefan Wermter
Due to the dynamic nature of human language, automatic speech recognition (ASR) systems need to continuously acquire new vocabulary. Out-Of-Vocabulary (OOV) words, such as trending…
Disentangling Prosody Representations with Unsupervised Speech Reconstruction
Leyuan Qu, Taihao Li, Cornelius Weber +3
Human speech can be characterized by different components, including semantic content, speaker identity and prosodic information. Significant progress has been made in disentanglin…
Improving Speech Emotion Recognition with Unsupervised Speaking Style Transfer
Leyuan Qu, Wei Wang, Cornelius Weber +3
Humans can effortlessly modify various prosodic attributes, such as the placement of stress and the intensity of sentiment, to convey a specific emotion while maintaining consisten…
A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learning
Gerald Schwiebert, Cornelius Weber, Leyuan Qu +2
Large datasets as required for deep learning of lip reading do not exist in many languages. In this paper we present the dataset GLips (German Lips) consisting of 250,000 publicly…
LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading
Leyuan Qu, Cornelius Weber, Stefan Wermter
The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of a…
Multimodal Target Speech Separation with Voice and Face References
Leyuan Qu, Cornelius Weber, Stefan Wermter
Target speech separation refers to isolating target speech from a multi-speaker mixture signal by conditioning on auxiliary information about the target speaker. Different from the…