132 citations · 182 across the 25 of their papers we have counts for
26 papers
Listening Deepfake Detection: A New Perspective Beyond Speaking-Centric Forgery Analysis
Miao Liu, Fangda Wei, Jing Wang +1
Existing deepfake detection research has primarily focused on scenarios where the manipulated subject is actively speaking, i.e., generating fabricated content by altering the spea…
NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
Liumeng Xue, Weizhen Bian, Jiahao Pan +9
Nonverbal vocalizations (NVVs), such as laughing, sighing, and sobbing, are essential for human-like speech, yet standardized evaluation rarely jointly assesses whether systems gen…
EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities
Xinyuan Qian, Xinjia Zhu, Alessio Brutti +1
TTM (Talking to Me) task is a pivotal component in understanding human social interactions, aiming to determine who is engaged in conversation with the camera-wearer. Traditional m…
Beyond Lips: Integrating Gesture and Lip Cues for Robust Audio-visual Speaker Extraction
Zexu Pan, Xinyuan Qian, Shengkui Zhao +2
Most audio-visual speaker extraction methods rely on synchronized lip recording to isolate the speech of a target speaker from a multi-talker mixture. However, in natural human com…
Analytic Incremental Learning For Sound Source Localization With Imbalance Rectification
Zexia Fan, Yu Chen, Qiquan Zhang +2
Sound source localization (SSL) demonstrates remarkable results in controlled settings but struggles in real-world deployment due to dual imbalance challenges: intra-task imbalance…
PerformSinger: Multimodal Singing Voice Synthesis Leveraging Synchronized Lip Cues from Singing Performance Videos
Ke Gu, Zhicong Wu, Peng Bai +5
Existing singing voice synthesis (SVS) models largely rely on fine-grained, phoneme-level durations, which limits their practical application. These methods overlook the complement…
AV-SSAN: Audio-Visual Selective DoA Estimation through Explicit Multi-Band Semantic-Spatial Alignment
Yu Chen, Hongxu Zhu, Jiadong Wang +2
Audio-visual sound source localization (AV-SSL) estimates the position of sound sources by fusing auditory and visual cues. Current AV-SSL methodologies typically require spatially…
Exploring Length Generalization For Transformer-based Speech Enhancement
Qiquan Zhang, Hongxu Zhu, Xinyuan Qian +2
Transformer network architecture has proven effective in speech enhancement. However, as its core module, self-attention suffers from quadratic complexity, making it infeasible for…
SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model
Xinyuan Qian, Jiaran Gao, Yaodan Zhang +4
Speech enhancement plays an essential role in various applications, and the integration of visual information has been demonstrated to bring substantial advantages. However, the ma…
Analytic Class Incremental Learning for Sound Source Localization with Privacy Protection
Xinyuan Qian, Xianghu Yue, Jiadong Wang +2
Sound Source Localization (SSL) enabling technology for applications such as surveillance and robotics. While traditional Signal Processing (SP)-based SSL methods provide analytic…
An Exploration of Length Generalization in Transformer-Based Speech Enhancement
Qiquan Zhang, Hongxu Zhu, Xinyuan Qian +2
The use of Transformer architectures has facilitated remarkable progress in speech enhancement. Training Transformers using substantially long speech utterances is often infeasible…
Mamba in Speech: Towards an Alternative to Self-Attention
Xiangyu Zhang, Qiquan Zhang, Hexin Liu +6
Transformer and its derivatives have achieved success in diverse tasks across computer vision, natural language processing, and speech processing. To reduce the complexity of compu…
Audio-Visual Target Speaker Extraction with Reverse Selective Auditory Attention
Ruijie Tao, Xinyuan Qian, Yidi Jiang +3
Audio-visual target speaker extraction (AV-TSE) aims to extract the specific person's speech from the audio mixture given auxiliary visual cues. Previous methods usually search for…
Enhancing Real-World Active Speaker Detection with Multi-Modal Extraction Pre-Training
Ruijie Tao, Xinyuan Qian, Rohan Kumar Das +3
Audio-visual active speaker detection (AV-ASD) aims to identify which visible face is speaking in a scene with one or more persons. Most existing AV-ASD methods prioritize capturin…
LocSelect: Target Speaker Localization with an Auditory Selective Hearing Mechanism
Yu Chen, Xinyuan Qian, Zexu Pan +2
The prevailing noise-resistant and reverberation-resistant localization algorithms primarily emphasize separating and providing directional output for each speaker in multi-speaker…
InterFormer: Interactive Local and Global Features Fusion for Automatic Speech Recognition
Zhi-Hao Lai, Tian-Hao Zhang, Qi Liu +5
The local and global features are both essential for automatic speech recognition (ASR). Many recent methods have verified that simply combining local and global features can furth…
Rethinking Speech Recognition with A Multimodal Perspective via Acoustic and Semantic Cooperative Decoding
Tian-Hao Zhang, Hai-Bo Qin, Zhi-Hao Lai +5
Attention-based encoder-decoder (AED) models have shown impressive performance in ASR. However, most existing AED methods neglect to simultaneously leverage both acoustic and seman…
Ripple sparse self-attention for monaural speech enhancement
Qiquan Zhang, Hongxu Zhu, Qi Song +3
The use of Transformer represents a recent success in speech enhancement. However, as its core component, self-attention suffers from quadratic complexity, which is computationally…
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Jiadong Wang, Xinyuan Qian, Malu Zhang +2
Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. The previous studies revealed the importan…
Predict-and-Update Network: Audio-Visual Speech Recognition Inspired by Human Speech Perception
Jiadong Wang, Xinyuan Qian, Haizhou Li
Audio and visual signals complement each other in human speech perception, so do they in speech recognition. The visual hint is less evident than the acoustic hint, but more robust…
Iterative Sound Source Localization for Unknown Number of Sources
Yanjie Fu, Meng Ge, Haoran Yin +4
Sound source localization aims to seek the direction of arrival (DOA) of all sound sources from the observed multi-channel audio. For the practical problem of unknown number of sou…
Speaker Extraction with Co-Speech Gestures Cue
Zexu Pan, Xinyuan Qian, Haizhou Li
Speaker extraction seeks to extract the clean speech of a target speaker from a multi-talker mixture speech. There have been studies to use a pre-recorded speech sample or face ima…
SLoClas: A Database for Joint Sound Localization and Classification
Xinyuan Qian, Bidisha Sharma, Amine El Abridi +1
In this work, we present the development of a new database, namely Sound Localization and Classification (SLoClas) corpus, for studying and analyzing sound localization and classif…
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Ruijie Tao, Zexu Pan, Rohan Kumar Das +3
Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and lo…