1 citations · 2 across the 22 of their papers we have counts for
23 papers
Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
Xuanru Zhou, Yiwen Shao, Jiahong Li +1
Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimi…
SoundscapeAgent: Agentic Soundscape Construction for Controllable Synthesis and Scalable Audio-Language Supervision
Hao Zhang, Yiwen Zhao, Yixuan Zhang +2
We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, a…
Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding
Zhiyuan Zhu, Yixuan Chen, Yiwen Shao +13
Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial rel…
Escape the Language Prior: Mitigating Late-Stage Modality Collapse in Audio Reasoning via Modality-Aware Policy Optimization
Cihan Xiao, Yiwen Shao, Chenxing Li +5
Audio and omni-modal large language models exhibit impressive cross-modal reasoning capabilities. However, standard reinforcement learning methods such as GRPO do not explicitly we…
Optimal physical approximations of pure-state cloning and transposition are complementary channels
Vanessa Brzić, Dmitry Grinko, Michał Studziński +1
State cloning and state transposition are fundamental transformations that, despite being desirable, cannot be perfectly realised as a consequence of two distinct requirements of q…
JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments
Zhan Liu, Changli Tang, Yuxin Wang +7
Current audio-visual large language models (AV-LLMs) are predominantly restricted to 2D perception, relying on RGB video and monaural audio. This design choice introduces a fundame…
SpatialEmb: Extract and Encode Spatial Information for 1-Stage Multi-channel Multi-speaker ASR on Arbitrary Microphone Arrays
Yiwen Shao, Yong Xu, Sanjeev Khudanpur +1
Spatial information is a critical clue for multi-channel multi-speaker target speech recognition. Most state-of-the-art multi-channel Automatic Speech Recognition (ASR) systems ext…
Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects
Kalvin Chang, Yiwen Shao, Jiahong Li +1
Despite having hundreds of millions of speakers, Chinese dialects lag behind Mandarin in speech and language technologies. Most varieties are primarily spoken, making dialect-to-Ma…
TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding
Mingyue Huo, Yiwen Shao, Yuheng Zhang
We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs:…
AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
Yiwen Shao, Wei Liu, Jiahong Li +4
Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a…
Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding
Mingyue Huo, Wei-Cheng Tseng, Yiwen Shao +2
Human voice encodes both identity and paralinguistic cues, yet encoders in large audio-language models (LALMs) rarely balance both aspects. In this work, we present a study toward…
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3
Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there is no consensus on whether audio-langu…
TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation
Wei Liu, Jiahong Li, Yiwen Shao +1
Speech-LLM models have demonstrated great performance in multi-modal and multi-task speech understanding. A typical speech-LLM paradigm is integrating speech modality with a large…
Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
Rongzhi Li, Ruogu Du, Zefang Chu +8
Serving Large Language Models (LLMs) is a GPU-intensive task where traditional autoscalers fall short, particularly for modern Prefill-Decode (P/D) disaggregated architectures. Thi…
Efficient Scaling for LLM-based ASR
Bingshen Mu, Yiwen Shao, Kun Wei +2
Large language model (LLM)-based automatic speech recognition (ASR) achieves strong performance but often incurs high computational costs. This work investigates how to obtain the…
DualSpeechLM: Towards Unified Speech Understanding and Generation via Dual Speech Token Modeling with Large Language Models
Yuanyuan Wang, Dongchao Yang, Yiwen Shao +5
Extending pre-trained text Large Language Models (LLMs)'s speech understanding or generation abilities by introducing various effective speech tokens has attracted great attention…
Efficient Multilingual ASR Finetuning via LoRA Language Experts
Jiahong Li, Yiwen Shao, Jianheng Zhuo +4
Recent advancements in deep learning have significantly enhanced multilingual automatic speech recognition (ASR) due to the development of advanced model architectures and availabl…
VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining
Jianheng Zhuo, Yifan Yang, Yiwen Shao +4
Automatic speech recognition (ASR) has made remarkable progress but heavily relies on large-scale labeled data, which is scarce for low-resource languages like Vietnamese. While ex…
Advancing Multi-talker ASR Performance with Large Language Models
Mohan Shi, Zengrui Jin, Yaoxun Xu +6
Recognizing overlapping speech from multiple speakers in conversational scenarios is one of the most challenging problem for automatic speech recognition (ASR). Serialized output t…
Multi-Channel Multi-Speaker ASR Using Target Speaker's Solo Segment
Yiwen Shao, Shi-Xiong Zhang, Yong Xu +4
In the field of multi-channel, multi-speaker Automatic Speech Recognition (ASR), the task of discerning and accurately transcribing a target speaker's speech within background nois…
RIR-SF: Room Impulse Response Based Spatial Feature for Target Speech Recognition in Multi-Channel Multi-Speaker Scenarios
Yiwen Shao, Shi-Xiong Zhang, Dong Yu
Automatic speech recognition (ASR) on multi-talker recordings is challenging. Current methods using 3D spatial data from multi-channel audio and visual cues focus mainly on direct…
UniX-Encoder: A Universal -Channel Speech Encoder for Ad-Hoc Microphone Array Speech Processing
Zili Huang, Yiwen Shao, Shi-Xiong Zhang +1
The speech field is evolving to solve more challenging scenarios, such as multi-channel recordings with multiple simultaneous talkers. Given the many types of microphone setups out…
Challenges and Insights: Exploring 3D Spatial Features and Complex Networks on the MISP Dataset
Yiwen Shao
Multi-channel multi-talker speech recognition presents formidable challenges in the realm of speech processing, marked by issues such as background noise, reverberation, and overla…