12 citations · 27 across the 12 of their papers we have counts for
12 papers
MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance
Xingjian Zhao, Zhe Xu, Qinyuan Cheng +20
Spoken dialogue systems often rely on cascaded pipelines that transcribe, process, and resynthesize speech. While effective, this design discards paralinguistic cues and limits exp…
Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM
Chenkun Tan, Pengyu Wang, Shaojun Zhou +6
Multimodal large language models (MLLMs) have gained significant attention due to their impressive ability to integrate vision and language modalities. Recent advancements in MLLMs…
MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
Mozhi Zhang, Pengyu Wang, Chenkun Tan +4
Large Language Models (LLMs) acquire extensive knowledge and remarkable abilities from extensive text corpora, making them powerful tools for various applications. To make LLMs mor…
IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities
Xin Zhang, Xiang Lyu, Zhihao Du +10
Current methods of building LLMs with voice interaction capabilities rely heavily on explicit text autoregressive generation before or during speech response generation to maintain…
SpeechAlign: Aligning Speech Generation to Human Preferences
Dong Zhang, Zhaowei Li, Shimin Li +4
Speech language models have significantly advanced in generating realistic speech, with neural codec language models standing out. However, the integration of human feedback to ali…
Calibrating the Confidence of Large Language Models by Eliciting Fidelity
Mozhi Zhang, Mianqiu Huang, Rundong Shi +5
Large language models optimized with techniques like RLHF have achieved good alignment in being helpful and harmless. However, post-alignment, these language models often exhibit o…
SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation
Dong Zhang, Xin Zhang, Jun Zhan +3
Benefiting from effective speech modeling, current Speech Large Language Models (SLLMs) have demonstrated exceptional capabilities in in-context speech generation and efficient gen…
SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems
Dong Zhang, Zhaowei Li, Pengyu Wang +3
Human communication is a complex and diverse process that not only involves multiple factors such as language, commonsense, and cultural backgrounds but also requires the participa…
SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
Xin Zhang, Dong Zhang, Shimin Li +2
Current speech large language models build upon discrete speech representations, which can be categorized into semantic tokens and acoustic tokens. However, existing speech tokens…
PromptNER: A Prompting Method for Few-shot Named Entity Recognition via k Nearest Neighbor Search
Mozhi Zhang, Hang Yan, Yaqian Zhou +1
Few-shot Named Entity Recognition (NER) is a task aiming to identify named entities via limited annotated samples. Recently, prototypical networks have shown promising performance…
DUB: Discrete Unit Back-translation for Speech Translation
Dong Zhang, Rong Ye, Tom Ko +2
How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT technique…
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
Dong Zhang, Shimin Li, Xin Zhang +4
Multi-modal large language models are regarded as a crucial step towards Artificial General Intelligence (AGI) and have garnered significant interest with the emergence of ChatGPT.…