18 papers
MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces
Guobin Ma, Yuepeng Jiang, Dake Guo +15
Continuous latent representations underlie recent diffusion-based lyrics-to-song and text-to-music systems, yet aligning lyric conditions with these latents remains difficult: exis…
StreamTN: A Low-Latency Streaming Chinese Text Normalization Model for Streaming TTS in Dialogue Systems
Wenhao Li, Jinrui Liang, Haoyu Zhang +11
Text-to-Speech (TTS) is an essential module that provides spoken responses in a spoken dialogue system (SDS) centered on a large language model (LLM). To ensure accurate TTS synthe…
Rethinking Music Tokenization: A Semantic Codec toward High-Fidelity LLM Music Generation
Huakang Chen, Guobin Ma, Yuepeng Jiang +10
Discrete audio tokenization has become the critical interface between raw waveforms and autoregressive modeling in recent music generation. As a result, music tokenizers must simul…
SphereVAE: Hyperspherical Latent Autoencoders for Robust Autoregressive Speech Representation Modeling
Haoyu Zhang, Jingbin Hu, Hanke Xie +8
With the rapid development of speech generation technology, discrete codec representations have been widely used because they provide a stable prediction paradigm. In expressive sp…
SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation
Hanke Xie, Haopeng Lin, Jiale Qian +13
Continuous-latent autoregressive speech generation has emerged as a promising alternative to discrete-token modeling by avoiding quantization loss and preserving richer acoustic in…
FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
Hanke Xie, Xiaming Ren, Dake Guo +10
Recent progress in speech dialogue systems requires Text-to-Speech (TTS) models to be faster and more responsive. Modern speech dialogue systems impose two primary requirements on…
MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6
Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streami…
SoulX-Transcriber: A Robust End-to-End Framework for Multi-Speaker Speech Transcription
Yuhang Dai, Haopeng Lin, Zhennan Lin +10
Recent advances in Automatic Speech Recognition (ASR) and Large Language Models (LLMs) have significantly improved speech understanding capabilities. However, multi-speaker speech…
MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
Huakang Chen, Jingbin Hu, Liumeng Xue +12
Instruction-following text-to-speech (TTS) has emerged as an important capability for controllable and expressive speech generation, yet its evaluation remains underdeveloped due t…
Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition
Yuhang Dai, Haopeng Lin, Jiale Qian +9
Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limi…
OmniCodec: Low Frame Rate Universal Audio Codec with Semantic-Acoustic Disentanglement
Jingbin Hu, Haoyu Zhang, Dake Guo +10
Large Language Models (LLMs) have advanced audio generation through discrete representation learning. However, most existing neural codecs focus on speech and emphasize reconstruct…
SoulX-Duplug: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation
Ruiqi Yan, Wenxi Chen, Zhanxun Liu +17
Recent advances in spoken dialogue systems have brought increased attention to human-like full-duplex voice interactions. However, our comprehensive review of this field reveals se…
SoulX-Singer: Towards High-Quality Zero-Shot Singing Voice Synthesis
Jiale Qian, Hao Meng, Tian Zheng +17
While recent years have witnessed rapid progress in speech synthesis, open-source singing voice synthesis (SVS) systems still face significant barriers to industrial deployment, pa…
SoulX-Podcast: Towards Realistic Long-form Podcasts with Dialectal and Paralinguistic Diversity
Hanke Xie, Haopeng Lin, Wenxiao Cao +17
Recent advances in text-to-speech (TTS) synthesis have significantly improved speech expressiveness and naturalness. However, most existing systems are tailored for single-speaker…
DialoSpeech: Dual-Speaker Dialogue Generation with LLM and Flow Matching
Hanke Xie, Dake Guo, Chengyou Wang +8
Recent advances in text-to-speech (TTS) synthesis, particularly those leveraging large language models (LLMs), have significantly improved expressiveness and naturalness. However,…
OSUM-EChat: Enhancing End-to-End Empathetic Spoken Chatbot via Understanding-Driven Spoken Dialogue
Xuelong Geng, Qijie Shao, Hongfei Xue +20
Empathy is crucial in enabling natural interactions within spoken dialogue systems, allowing machines to recognize and respond appropriately to paralinguistic cues such as age, gen…
SenSE: Semantic-Aware High-Fidelity Universal Speech Enhancement
Xingchen Li, Hanke Xie, Ziqian Wang +4
Generative Universal Speech Enhancement (USE) methods aim to leverage generative models to improve speech quality under various types of distortions. However, existing generative s…
Llasa+: Free Lunch for Accelerated and Streaming Llama-Based Speech Synthesis
Wenjie Tian, Xinfa Zhu, Hanke Xie +3
Recent progress in text-to-speech (TTS) has achieved impressive naturalness and flexibility, especially with the development of large language model (LLM)-based approaches. However…