2 citations · 9 across the 30 of their papers we have counts for
30 papers
DialoSpeech: Dual-Speaker Dialogue Generation with LLM and Flow Matching
Hanke Xie, Dake Guo, Chengyou Wang +8
Recent advances in text-to-speech (TTS) synthesis, particularly those leveraging large language models (LLMs), have significantly improved expressiveness and naturalness. However,…
Qwen3-Omni Technical Report
Jin Xu, Zhifang Guo, Hangrui Hu +35
We present Qwen3-Omni, a single multimodal model that, for the first time, maintains state-of-the-art performance across text, image, audio, and video without any degradation relat…
MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech
Kangxiang Xia, Xinfa Zhu, Jixun Yao +1
In recent years, text-to-speech (TTS) has seen impressive advancements through large-scale language models, achieving human-level speech quality. Integrating human feedback has pro…
XEmoRAG: Cross-Lingual Emotion Transfer with Controllable Intensity Using Retrieval-Augmented Generation
Tianlun Zuo, Jingbin Hu, Yuke Li +6
Zero-shot emotion transfer in cross-lingual speech synthesis refers to generating speech in a target language, where the emotion is expressed based on reference speech from a diffe…
DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis
Wenjie Tian, Xinfa Zhu, Haohe Liu +6
While recent video-to-audio (V2A) models can generate realistic background audio from visual input, they largely overlook speech, an essential part of many video soundtracks. This…
Weakly Supervised Data Refinement and Flexible Sequence Compression for Efficient Thai LLM-based ASR
Mingchen Shao, Xinfa Zhu, Chengyou Wang +6
Despite remarkable achievements, automatic speech recognition (ASR) in low-resource scenarios still faces two challenges: high-quality data scarcity and high computational demands.…
FlowSE: Efficient and High-Quality Speech Enhancement via Flow Matching
Ziqian Wang, Zikai Liu, Xinfa Zhu +6
Generative models have excelled in audio tasks using approaches such as language models, diffusion, and flow matching. However, existing generative approaches for speech enhancemen…
U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding
Ziqian Wang, Xianjun Xia, Xinfa Zhu +1
The text generation paradigm for audio tasks has opened new possibilities for unified audio understanding. However, existing models face significant challenges in achieving a compr…
Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokens
Xinsheng Wang, Mingqi Jiang, Ziyang Ma +22
Recent advancements in large language models (LLMs) have driven significant progress in zero-shot text-to-speech (TTS) synthesis. However, existing foundation models rely on multi-…
LLaSE-G1: Incentivizing Generalization Capability for LLaMA-based Speech Enhancement
Boyi Kang, Xinfa Zhu, Zihan Zhang +10
Recent advancements in language models (LMs) have demonstrated strong capabilities in semantic understanding and contextual modeling, which have flourished in generative speech enh…
OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia
Xuelong Geng, Kun Wei, Qijie Shao +18
Large Language Models (LLMs) have made significant progress in various downstream tasks, inspiring the development of Speech Understanding Language Models (SULMs) to enable compreh…
Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thought
Zhixian Zhao, Xinfa Zhu, Xinsheng Wang +4
Large-scale audio language models (ALMs), such as Qwen2-Audio, are capable of comprehending diverse audio signal, performing audio analysis and generating textual responses. Howeve…
YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controls
Zihao Chen, Haomin Zhang, Xinhan Di +10
Generating sound effects for product-level videos, where only a small amount of labeled data is available for diverse scenes, requires the production of high-quality sounds in few-…
CoDiff-VC: A Codec-Assisted Diffusion Model for Zero-shot Voice Conversion
Yuke Li, Xinfa Zhu, Hanzhao Li +6
Zero-shot voice conversion (VC) aims to convert the original speaker's timbre to any target speaker while keeping the linguistic content. Current mainstream zero-shot voice convers…
The NPU-HWC System for the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge
Dake Guo, Jixun Yao, Xinfa Zhu +6
This paper presents the NPU-HWC system submitted to the ISCSLP 2024 Inspirational and Convincing Audio Generation Challenge 2024 (ICAGC). Our system consists of two modules: a spee…
Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy
Linhan Ma, Xinfa Zhu, Yuanjun Lv +5
Zero-shot voice conversion (VC) aims to transform source speech into arbitrary unseen target voice while keeping the linguistic content unchanged. Recent VC methods have made signi…
Text-aware and Context-aware Expressive Audiobook Speech Synthesis
Dake Guo, Xinfa Zhu, Liumeng Xue +3
Recent advances in text-to-speech have significantly improved the expressiveness of synthetic speech. However, a major challenge remains in generating speech that captures the dive…
Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
Hanzhao Li, Liumeng Xue, Haohan Guo +6
The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid t…
SELM: Speech Enhancement Using Discrete Tokens and Language Models
Ziqian Wang, Xinfa Zhu, Zihan Zhang +4
Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the in…
Accent-VITS:accent transfer for end-to-end TTS
Linhan Ma, Yongmao Zhang, Xinfa Zhu +4
Accent transfer aims to transfer an accent from a source speaker to synthetic speech in the target speaker's voice. The main challenge is how to effectively disentangle speaker tim…
SponTTS: modeling and transferring spontaneous style for TTS
Hanzhao Li, Xinfa Zhu, Liumeng Xue +3
Spontaneous speaking style exhibits notable differences from other speaking styles due to various spontaneous phenomena (e.g., filled pauses, prolongation) and substantial prosody…
Vec-Tok Speech: speech vectorization and tokenization for neural speech generation
Xinfa Zhu, Yuanjun Lv, Yi Lei +5
Language models (LMs) have recently flourished in natural language processing and computer vision, generating high-fidelity texts or images in various tasks. In contrast, the curre…
HiGNN-TTS: Hierarchical Prosody Modeling with Graph Neural Networks for Expressive Long-form TTS
Dake Guo, Xinfa Zhu, Liumeng Xue +4
Recent advances in text-to-speech, particularly those based on Graph Neural Networks (GNNs), have significantly improved the expressiveness of short-form synthetic speech. However,…
U-Style: Cascading U-nets with Multi-level Speaker and Style Modeling for Zero-Shot Voice Cloning
Tao Li, Zhichao Wang, Xinfa Zhu +4
Zero-shot speaker cloning aims to synthesize speech for any target speaker unseen during TTS system building, given only a single speech reference of the speaker at hand. Although…