3 citations · 5 across the 16 of their papers we have counts for
18 papers
Multimodal Target Speaker Extraction: Towards Unified Speaker Cues Across Modalities
Xinyuan Qian, Yanghao Zhou, Ziyang Jiang +10
Target Speaker Extraction (TSE) is pivotal in speech communication and human-computer interaction, enabling the isolation of a specific speaker's voice from complex acoustic enviro…
HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models
Feiyu Zhao, Yiming Chen, Wenhuan Lu +3
Large Audio-Language Models (LALMs) have recently achieved strong performance across various audio-centric tasks. However, hallucination, where models generate responses that are s…
Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion
Sen Zhang, Jianguo Wei, Wenhuan Lu +6
The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in si…
Listening for "You": Enhancing Speech Image Retrieval via Target Speaker Extraction
Wenhao Yang, Jianguo Wei, Wenhuan Lu +2
Image retrieval using spoken language cues has emerged as a promising direction in multimodal perception, yet leveraging speech in multi-speaker scenarios remains challenging. We p…
Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculture
Meng Cui, Xianghu Yue, Xinyuan Qian +5
Fish Feeding Intensity Assessment (FFIA) is crucial in industrial aquaculture management. Recent multi-modal approaches have shown promise in improving FFIA robustness and efficien…
UniCodec: Unified Audio Codec with Single Domain-Adaptive Codebook
Yidi Jiang, Qian Chen, Shengpeng Ji +6
The emergence of audio language models is empowered by neural audio codecs, which establish critical mappings between continuous waveforms and discrete tokens compatible with langu…
PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning
Xianghu Yue, Yiming Chen, Xueyi Zhang +5
Multi-modal class-incremental learning (MMCIL) seeks to leverage multi-modal data, such as audio-visual and image-text pairs, thereby enabling models to learn continuously across a…
VoiceBench: Benchmarking LLM-Based Voice Assistants
Yiming Chen, Xianghu Yue, Chen Zhang +3
Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering…
Analytic Class Incremental Learning for Sound Source Localization with Privacy Protection
Xinyuan Qian, Xianghu Yue, Jiadong Wang +2
Sound Source Localization (SSL) enabling technology for applications such as surveillance and robotics. While traditional Signal Processing (SP)-based SSL methods provide analytic…
Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models
Yiming Chen, Xianghu Yue, Xiaoxue Gao +4
Various audio-LLMs (ALLMs) have been explored recently for tackling different audio tasks simultaneously using a single, unified model. While existing evaluations of ALLMs primaril…
TTSlow: Slow Down Text-to-Speech with Efficiency Robustness Evaluations
Xiaoxue Gao, Yiming Chen, Xianghu Yue +2
Text-to-speech (TTS) has been extensively studied for generating high-quality speech with textual inputs, playing a crucial role in various real-time applications. For real-world d…
CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing
Xianghu Yue, Xiaohai Tian, Lu Lu +3
There has been a long-standing quest for a unified audio-visual-text model to enable various multimodal understanding tasks, which mimics the listening, seeing and reading process…
Text-guided HuBERT: Self-Supervised Speech Pre-training via Generative Adversarial Networks
Duo Ma, Xianghu Yue, Junyi Ao +2
Human language can be expressed in either written or spoken form, i.e. text or speech. Humans can acquire knowledge from text to improve speaking and listening. However, the quest…
Self-Supervised Acoustic Word Embedding Learning via Correspondence Transformer Encoder
Jingru Lin, Xianghu Yue, Junyi Ao +1
Acoustic word embeddings (AWEs) aims to map a variable-length speech segment into a fixed-dimensional representation. High-quality AWEs should be invariant to variations, such as d…
Self-Transriber: Few-shot Lyrics Transcription with Self-training
Xiaoxue Gao, Xianghu Yue, Haizhou Li
The current lyrics transcription approaches heavily rely on supervised learning with labeled data, but such data are scarce and manual labeling of singing is expensive. How to bene…
token2vec: A Joint Self-Supervised Pre-training Framework Using Unpaired Speech and Text
Xianghu Yue, Junyi Ao, Xiaoxue Gao +1
Self-supervised pre-training has been successful in both text and speech processing. Speech and text offer different but complementary information. The question is whether we are a…
End-to-End Code-Switching ASR for Low-Resourced Language Pairs
Xianghu Yue, Grandee Lee, Emre Yılmaz +2
Despite the significant progress in end-to-end (E2E) automatic speech recognition (ASR), E2E ASR for low resourced code-switching (CS) speech has not been well studied. In this wor…
Multi-Graph Decoding for Code-Switching ASR
Emre Yılmaz, Samuel Cohen, Xianghu Yue +2
In the FAME! Project, a code-switching (CS) automatic speech recognition (ASR) system for Frisian-Dutch speech is developed that can accurately transcribe the local broadcaster's b…