1 citations · 4 across the 11 of their papers we have counts for
13 papers
Full-Duplex Interaction with Speech, Expressions and Gestures
Siyin Wang, Wenyi Yu, Terumi Chiba +5
Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue mo…
SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents
Wenyi Yu, Siyin Wang, Terumi Chiba +5
Full-duplex speech large language models (LLMs) enable low-latency, natural voice interaction. However, real-world agents must also use tools and perform deliberative reasoning-ope…
FlexiSLM: A Spoken Language Model with Dynamic and Controllable Frame Rates
Jiaqi Li, Chaoren Wang, Xiaohai Tian +9
Spoken language models (SLMs) extend LLMs to speech input and output, but existing systems use fixed frame rates (e.g., 25 or 12.5 Hz), overlooking speech's time-varying informatio…
End-to-end Listen, Look, Speak and Act
Siyin Wang, Wenyi Yu, Xianzhao Chen +4
Human interaction is inherently multimodal and full-duplex: we listen while watching, speak while acting, and fluidly adapt to turn-taking and interruptions. Realizing these capabi…
SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation
Wenyi Yu, Siyin Wang, Xiaoyu Yang +7
In order to enable fluid and natural human-machine speech interaction, existing full-duplex conversational systems often adopt modular architectures with auxiliary components such…
Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context
Junyi Ao, Dekun Chen, Xiaohai Tian +6
Large Language Models (LLMs) have recently shown remarkable ability to process not only text but also multimodal inputs such as speech and audio. However, most existing models prim…
QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions
Siyin Wang, Wenyi Yu, Xianzhao Chen +7
This paper explores a novel perspective to speech quality assessment by leveraging natural language descriptions, offering richer, more nuanced insights than traditional numerical…
SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation
Wenyi Yu, Siyin Wang, Xiaoyu Yang +7
Full-duplex multimodal large language models (LLMs) provide a unified framework for addressing diverse speech understanding and generation tasks, enabling more natural and seamless…
Enabling Auditory Large Language Models for Automatic Speech Quality Evaluation
Siyin Wang, Wenyi Yu, Yudong Yang +10
Speech quality assessment typically requires evaluating audio from multiple aspects, such as mean opinion score (MOS) and speaker similarity (SIM) \etc., which can be challenging t…
A Comprehensive Solution to Connect Speech Encoder and Large Language Model for ASR
Van Tung Pham, Yist Lin, Tao Han +4
Recent works have shown promising results in connecting speech encoders to large language models (LLMs) for speech recognition. However, several limitations persist, including limi…
Can Large Language Models Understand Spatial Audio?
Changli Tang, Wenyi Yu, Guangzhi Sun +8
This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMs'…
Dual-Pipeline with Low-Rank Adaptation for New Language Integration in Multilingual ASR
Yerbolat Khassanov, Zhipeng Chen, Tianfeng Chen +5
This paper addresses challenges in integrating new languages into a pre-trained multilingual automatic speech recognition (mASR) system, particularly in scenarios where training da…
SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words
Junyi Ao, Yuancheng Wang, Xiaohai Tian +6
Speech encompasses a wealth of information, including but not limited to content, paralinguistic, and environmental information. This comprehensive nature of speech significantly i…