9 papers
Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
Xuanru Zhou, Yiwen Shao, Jiahong Li +1
Multimodal large language models (MLLMs) are typically built through a multi-stage pipeline consisting of cross-modal alignment, supervised fine-tuning (SFT), and preference optimi…
Optimal physical approximations of pure-state cloning and transposition are complementary channels
Vanessa Brzić, Dmitry Grinko, Michał Studziński +1
State cloning and state transposition are fundamental transformations that, despite being desirable, cannot be perfectly realised as a consequence of two distinct requirements of q…
Towards Comprehensive Semantic Speech Embeddings for Chinese Dialects
Kalvin Chang, Yiwen Shao, Jiahong Li +1
Despite having hundreds of millions of speakers, Chinese dialects lag behind Mandarin in speech and language technologies. Most varieties are primarily spoken, making dialect-to-Ma…
AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
Yiwen Shao, Wei Liu, Jiahong Li +4
Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a…
Auden-Voice: General-Purpose Voice Encoder for Speech and Language Understanding
Mingyue Huo, Wei-Cheng Tseng, Yiwen Shao +2
Human voice encodes both identity and paralinguistic cues, yet encoders in large audio-language models (LALMs) rarely balance both aspects. In this work, we present a study toward…
Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation
Wei-Cheng Tseng, Xuanru Zhou, Mingyue Huo +3
Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there is no consensus on whether audio-langu…
TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation
Wei Liu, Jiahong Li, Yiwen Shao +1
Speech-LLM models have demonstrated great performance in multi-modal and multi-task speech understanding. A typical speech-LLM paradigm is integrating speech modality with a large…
Efficient Scaling for LLM-based ASR
Bingshen Mu, Yiwen Shao, Kun Wei +2
Large language model (LLM)-based automatic speech recognition (ASR) achieves strong performance but often incurs high computational costs. This work investigates how to obtain the…
Efficient Multilingual ASR Finetuning via LoRA Language Experts
Jiahong Li, Yiwen Shao, Jianheng Zhuo +4
Recent advancements in deep learning have significantly enhanced multilingual automatic speech recognition (ASR) due to the development of advanced model architectures and availabl…