132 citations · 189 across the 21 of their papers we have counts for
23 papers
Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities
Xinyuan Qian, Yanghao Zhou, Ziyang Jiang +10
Target Speaker Extraction (TSE) is pivotal in speech communication and human-computer interaction, enabling the isolation of a specific speaker's voice from complex acoustic enviro…
SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex Scenarios
Ziyang Jiang, Yu Chen, Zexu Pan +5
Humans can selectively attend to a target sound and estimate its direction in complex scenarios, whereas such selective localization remains challenging for current deep learning-b…
MeMo: Attentional Momentum for Real-Time Audio-Visual Target Speaker Extraction Under Impaired Visual Conditions
Junjie Li, Wenxuan Wu, Shuai Wang +4
Audio-visual Target Speaker Extraction (AV-TSE) aims to isolate a target speaker's voice from multi-speaker environments by leveraging visual cues as guidance. However, the perform…
Causal Self-supervised Pretrained Frontend with Predictive Code for Speech Separation
Wupeng Wang, Zexu Pan, Xinke Li +2
Speech separation (SS) seeks to disentangle a multi-talker speech mixture into single-talker speech streams. Although SS can be generally achieved using offline methods, such a pro…
Context-Aware Two-Step Training Scheme for Domain Invariant Speech Separation
Wupeng Wang, Zexu Pan, Jingru Lin +2
Speech separation seeks to isolate individual speech signals from a multi-talk speech mixture. Despite much progress, a system well-trained on synthetic data often experiences perf…
Speech Separation with Pretrained Frontend to Minimize Domain Mismatch
Wupeng Wang, Zexu Pan, Xinke Li +2
Speech separation seeks to separate individual speech signals from a speech mixture. Typically, most separation models are trained on synthetic data due to the unavailability of ta…
pTSE-T: Presentation Target Speaker Extraction using Unaligned Text Cues
Ziyang Jiang, Jiahe Lei, Xueyan Chen +4
Target Speaker Extraction (TSE) aims to extract the clean speech of the target speaker in an audio mixture, eliminating irrelevant background noise and speech. While prior work has…
LocSelect: Target Speaker Localization with an Auditory Selective Hearing Mechanism
Yu Chen, Xinyuan Qian, Zexu Pan +2
The prevailing noise-resistant and reverberation-resistant localization algorithms primarily emphasize separating and providing directional output for each speaker in multi-speaker…
Audio-Visual Active Speaker Extraction for Sparsely Overlapped Multi-talker Speech
Junjie Li, Ruijie Tao, Zexu Pan +3
Target speaker extraction aims to extract the speech of a specific speaker from a multi-talker mixture as specified by an auxiliary reference. Most studies focus on the scenario wh…
NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals
Zexu Pan, Marvin Borsdorf, Siqi Cai +2
Humans possess the remarkable ability to selectively attend to a single speaker amidst competing voices and background noise, known as selective auditory attention. Recent studies…
Target Active Speaker Detection with Audio-visual Cues
Yidi Jiang, Ruijie Tao, Zexu Pan +1
In active speaker detection (ASD), we would like to detect whether an on-screen person is speaking based on audio-visual cues. Previous studies have primarily focused on modeling a…
Rethinking the visual cues in audio-visual speaker extraction
Junjie Li, Meng Ge, Zexu pan +4
The Audio-Visual Speaker Extraction (AVSE) algorithm employs parallel video recording to leverage two visual cues, namely speaker identity and synchronization, to enhance performan…
Late Audio-Visual Fusion for In-The-Wild Speaker Diarization
Zexu Pan, Gordon Wichern, François G. Germain +2
Speaker diarization is well studied for constrained audios but little explored for challenging in-the-wild videos, which have more speakers, shorter utterances, and inconsistent on…
ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting
Zexu Pan, Wupeng Wang, Marvin Borsdorf +1
The speaker extraction technique seeks to single out the voice of a target speaker from the interfering voices in a speech mixture. Typically an auxiliary reference of the target s…
VCSE: Time-Domain Visual-Contextual Speaker Extraction Network
Junjie Li, Meng Ge, Zexu Pan +2
Speaker extraction seeks to extract the target speech in a multi-talker scenario given an auxiliary reference. Such reference can be auditory, i.e., a pre-recorded speech, visual,…
A Hybrid Continuity Loss to Reduce Over-Suppression for Time-domain Target Speaker Extraction
Zexu Pan, Meng Ge, Haizhou Li
The speaker extraction algorithm extracts the target speech from a mixture speech containing interference speech and background noise. The extraction process sometimes over-suppres…
Speaker Extraction with Co-Speech Gestures Cue
Zexu Pan, Xinyuan Qian, Haizhou Li
Speaker extraction seeks to extract the clean speech of a target speaker from a multi-talker mixture speech. There have been studies to use a pre-recorded speech sample or face ima…
USEV: Universal Speaker Extraction with Visual Cue
Zexu Pan, Meng Ge, Haizhou Li
A speaker extraction algorithm seeks to extract the target speaker's speech from a multi-talker speech mixture. The prior studies focus mostly on speaker extraction from a highly o…
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Ruijie Tao, Zexu Pan, Rohan Kumar Das +3
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and lo…
Selective Listening by Synchronizing Speech with Lips
Zexu Pan, Ruijie Tao, Chenglin Xu +1
A speaker extraction algorithm seeks to extract the speech of a target speaker from a multi-talker speech mixture when given a cue that represents the target speaker, such as a pre…
Multi-target DoA Estimation with an Audio-visual Fusion Mechanism
Xinyuan Qian, Maulik Madhavi, Zexu Pan +2
Most of the prior studies in the spatial \ac{DoA} domain focus on a single modality. However, humans use auditory and visual senses to detect the presence of sound sources. With th…
Muse: Multi-modal target speaker extraction with visual cues
Zexu Pan, Ruijie Tao, Chenglin Xu +1
Speaker extraction algorithm relies on the speech sample from the target speaker as the reference point to focus its attention. Such a reference speech is typically pre-recorded. O…
Multi-modal Attention for Speech Emotion Recognition
Zexu Pan, Zhaojie Luo, Jichen Yang +1
Emotion represents an essential aspect of human speech that is manifested in speech prosody. Speech, visual, and textual cues are complementary in human communication. In this pape…