23 papers
DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?
Jiaheng Dong, Shihong Tan, Gongping Huang +3
End-to-end (E2E) full-duplex spoken language models (FD-SLMs) enable simutanous listening and speaking continously, requiring models to take appropriate interaction actions as conv…
VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models
Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden +1
Spoken conversational systems must recover information from prior interactions (i.e., memory), yet relevant information in speech extends beyond what was said to who said it, how i…
When Intent Arrives Late: A Benchmark for Full-Duplex Speech Models under Delayed Intent Revelation
Yang Xiao, Tianyi Peng, Hanyu Meng +1
Native full-duplex speech models can respond before a user finishes speaking, making the behavior of the model depend on both the final utterance and when intent-defining informati…
Exemplar-Free Analytic Learning for Multi-Label Audio Class-Incremental Learning
Siyuan Luo, Yang Xiao, Ting Dang
Audio classification is inherently a multi-label task, as real-world acoustic environments contain multiple simultaneous sound events. When new sound classes emerge, models must in…
Easper: An Accessible ASR Pipeline for Language Documentation
Aso Mahmudi, Ting Dang, Ekaterina Vylomova +1
Audio transcription is a critical bottleneck in language documentation. While multilingual Automatic Speech Recognition (ASR) models like Whisper offer solutions, field linguists o…
MADBench: A Benchmark for Modality-Aware Audio Deepfake Detection
Yanqiu Li, Yang Xiao, Jisheng Bai +3
Recent advances in speech synthesis and audio generation have made high-fidelity acoustic forgery low-cost and difficult to attribute, enabling a realistic attack scenario in which…
QuaSR: Quality-Aware Sample Reweighting for Pacific Indigenous Speech Recognition
Yishun Li, Yang Xiao, Gongping Huang +3
Training automatic speech recognition (ASR) models for low-resource languages is challenging due to limited data and highly variable supervision quality. In particular, Pacific Ind…
Overview of ESDD2: Environment-Aware Speech and Sound Deepfake Detection Challenge
Xueping Zhang, Han Yin, Yang Xiao +4
The Environment-Aware Speech and Sound Deepfake Detection Challenge (ESDD2), held in conjunction with ICME 2026, evaluated systems for five component-level audio spoofing detection…
RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
Hongyu Jin, Siyi Wang, Yang Xiao +11
Humans process rich auditory environments through various cognitive capabilities such as audio perception, audio reasoning and memory. Despite recent progress in large audio-langua…
Rethinking Continual Learning for Speech and Audio: A Representation-Centric Taxonomy and Open Problems
Yang Xiao, Siyi Wang, Eun-Jung Holden +1
Speech and audio systems operate in inherently non-stationary environments, yet continual learning (CL) research in this domain, especially in the foundation model era, remains fra…
Why Can't They Remember? Uncovering Representation and Retrieval Bottlenecks in Multi-Turn Acoustic Memory
Yang Xiao, Siyi Wang, Han Yin +4
Large audio language models (LALMs) process both speech and environmental acoustic cues, yet struggle to retain non-speech information across multi-turn interactions. The performan…
A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
Kaiwen Luo, Zhenhong Zhou, Leyan Wang +32
Advances in Large Language Models (LLMs) have paved the way for Multimodal Large Language Models (MLLMs). Among these, Large Audio Language Models (LALMs) are essential for realizi…
Continual Adaptation for Pacific Indigenous Speech Recognition
Yang Xiao, Aso Mahmudi, Nick Thieberger +3
Speech foundation models struggle with low-resource Pacific Indigenous languages because of severe data scarcity. Furthermore, full fine-tuning risks catastrophic forgetting. To ad…
ImKWS: Test-Time Adaptation for Keyword Spotting with Class Imbalance
Hanyu Ding, Yang Xiao, Jiaheng Dong +1
Keyword spotting (KWS) identifies words for voice assistants, but environmental noise frequently reduces accuracy. Standard adaptation fixes this issue and strictly requires origin…
Activation Steering for Accent Adaptation in Large Audio Language Models
Jinuo Sun, Yang Xiao, Sung Kyun Chung +4
Accent variability remains a major source of errors in automatic speech recognition, yet most adaptation methods rely on parameter fine-tuning without understanding where accent in…
PolyBench: A Benchmark for Compositional Reasoning in Polyphonic Audio
Yuanjian Chen, Yang Xiao, Han Yin +3
Large Audio Language Models (LALMs) are increasingly capable of reasoning over audio, yet existing benchmarks offer limited coverage of reasoning in polyphonic audio, where multipl…
The First Environmental Sound Deepfake Detection Challenge: Benchmarking Robustness, Evaluation, and Insights
Han Yin, Yang Xiao, Rohan Kumar Das +2
Recent progress in audio generation has made it increasingly easy to create highly realistic environmental soundscapes, which can be misused to produce deceptive content, such as f…
Focus Then Listen: An Empirical Study of Plug-and-Play Audio Enhancer for Noise-Robust Large Audio Language Models
Han Yin, Yang Xiao, Younghoo Kwon +2
Large audio language models (LALMs) are a class of foundation models for audio understanding. Existing LALMs tend to degrade significantly in real-world noisy acoustic conditions w…
ESDD2: Environment-Aware Speech and Sound Deepfake Detection Challenge Evaluation Plan
Xueping Zhang, Han Yin, Yang Xiao +4
Audio recorded in real-world environments often contains a mixture of foreground speech and background environmental sounds. With rapid advances in text-to-speech, voice conversion…
Adapting Where It Matters: Depth-Aware Adaptation for Efficient Multilingual Speech Recognition in Low-Resource Languages
Yang Xiao, Eun-Jung Holden, Ting Dang
Recent speech foundation models excel at multilingual automatic speech recognition (ASR) for high-resource languages, but adapting them to low-resource languages remains challengin…
Environmental Sound Deepfake Detection Challenge: An Overview
Han Yin, Yang Xiao, Rohan Kumar Das +2
Recent progress in audio generation models has made it possible to create highly realistic and immersive soundscapes, which are now widely used in film and virtual-reality-related…
ESDD 2026: Environmental Sound Deepfake Detection Challenge Evaluation Plan
Han Yin, Yang Xiao, Rohan Kumar Das +2
Recent advances in audio generation systems have enabled the creation of highly realistic and immersive soundscapes, which are increasingly used in film and virtual reality. Howeve…
RawTFNet: A Lightweight CNN Architecture for Speech Anti-spoofing
Yang Xiao, Ting Dang, Rohan Kumar Das
Automatic speaker verification (ASV) systems are often affected by spoofing attacks. Recent transformer-based models have improved anti-spoofing performance by learning strong feat…