28 papers
MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces
Guobin Ma, Yuepeng Jiang, Dake Guo +15
Continuous latent representations underlie recent diffusion-based lyrics-to-song and text-to-music systems, yet aligning lyric conditions with these latents remains difficult: exis…
The NVVSpeech Challenge 2026: Overview and Lessons for Speech Understanding and Generation with Nonverbal Vocalizations
Liumeng Xue, Yuang Cao, Jingbin Hu +12
Nonverbal vocalizations (NVVs), such as laughter and sighs, convey meaning alongside words. NVV-aware speech systems must recognize and generate these events while preserving lexic…
WenetSpeech-Min: A Large-Scale Minnan Speech Corpus with Dual Transcriptions for Dialectal Speech Processing
Haoyu Zhang, Chunjiang He, Hongtao Li +17
Progress in dialectal speech technology is hindered by the scarcity of large-scale, real-world corpora. For Minnan speech, existing resources remain limited, and few provide paired…
Interactive TTS: Dynamic Speaking Style Adaptation for Expressive Speech Synthesis
Wenjie Tian, Kangxiang Xia, Jingbin Hu +7
Dynamic speaking style adaptation in multi-turn multimodal interaction remains a major challenge for text-to-speech (TTS) systems. Existing context-aware TTS (CTTS) methods typical…
StreamTN: A Low-Latency Streaming Chinese Text Normalization Model for Streaming TTS in Dialogue Systems
Wenhao Li, Jinrui Liang, Haoyu Zhang +11
Text-to-Speech (TTS) is an essential module that provides spoken responses in a spoken dialogue system (SDS) centered on a large language model (LLM). To ensure accurate TTS synthe…
Rethinking Music Tokenization: A Semantic Codec toward High-Fidelity LLM Music Generation
Huakang Chen, Guobin Ma, Yuepeng Jiang +10
Discrete audio tokenization has become the critical interface between raw waveforms and autoregressive modeling in recent music generation. As a result, music tokenizers must simul…
Preference Optimization with LALM Feedback for Continuous Autoregressive Non-Verbal Vocalization Generation
Jingbin Hu, Qirui Zhan, Yuang Cao +8
We propose a preference optimization framework with Large Audio-Language Model (LALM) feedback for controllable non-verbal vocalization (NVV) generation in continuous autoregressiv…
SpeechAnnotator: A Context-Aware Multi-Agent Framework and Benchmark for Multidimensional Speech Annotation
Qirui Zhan, Shuiyuan Wang, Jingbin Hu +10
Recent controllable speech generation requires training data with fine-grained annotations of speaker traits, prosody, emotion, paralinguistic cues, acoustic scenes, and context. E…
Source-Adaptive Data Curation for Bilingual NVV-Aware ASR
Yuang Cao, Qirui Zhan, Jingbin Hu +8
Nonverbal vocalizations (NVVs), such as laughter, sighs, breaths, and coughs, convey affective and interactional information that conventional automatic speech recognition (ASR) sy…
SphereVAE: Hyperspherical Latent Autoencoders for Robust Autoregressive Speech Representation Modeling
Haoyu Zhang, Jingbin Hu, Hanke Xie +8
With the rapid development of speech generation technology, discrete codec representations have been widely used because they provide a stable prediction paradigm. In expressive sp…
SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation
Hanke Xie, Haopeng Lin, Jiale Qian +13
Continuous-latent autoregressive speech generation has emerged as a promising alternative to discrete-token modeling by avoiding quantization loss and preserving richer acoustic in…
Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation
Ziyu Zhang, Chunyu Qiang, Xiaopeng Wang +10
While song generation and singing voice conversion (SVC) have evolved significantly, they have long been developed isolated: the former lacks zero-shot speaker cloning, while the l…
Beyond Semantic Dominance: Cognitive Affective Reasoning and Empathetic Response Alignment in Audio Language Models
Zhixian Zhao, Shuiyuan Wang, Wenjie Tian +3
While Audio Language Models (ALMs) demonstrate strong semantic understanding, they struggle with complex affective interactions. Specifically, textual semantic dominance often over…
FlashTTS: Fast Streaming TTS with MTP Acceleration and X-pred Mean Flow Distillation
Hanke Xie, Xiaming Ren, Dake Guo +10
Recent progress in speech dialogue systems requires Text-to-Speech (TTS) models to be faster and more responsive. Modern speech dialogue systems impose two primary requirements on…
MeanVC 2: Robust Low-Latency Streaming Zero-Shot Voice Conversion
Guobin Ma, Yuxuan Xia, Yuepeng Jiang +6
Streaming zero-shot voice conversion (VC) has become increasingly popular due to its potential for real-time applications. The recently proposed MeanVC achieves lightweight streami…
UrduSpeech: A 156-Hour Urdu Speech Corpus with 12-Dimension Paralinguistic Annotations
Attia Nafees ul Haq, Zeyu Zhu, Jingbin Hu +2
Despite 230 million speakers, Urdu remains critically under-resourced in speech technology. We introduce UrduSpeech: a large high-fidelity Urdu corpus comprising 156 hours of audio…
Towards Fine-Grained Multi-Dimensional Speech Understanding: Data Pipeline, Benchmark, and Model
Guojian Li, Zhixian Zhao, Zhennan Lin +9
While speech Large Language Models (LLMs) excel at conventional tasks like basic speech recognition, they lack fine-grained, multi-dimensional perception. This deficiency is eviden…
MINT-Bench: A Comprehensive Multilingual Benchmark for Instruction-Following Text-to-Speech
Huakang Chen, Jingbin Hu, Liumeng Xue +12
Instruction-following text-to-speech (TTS) has emerged as an important capability for controllable and expressive speech generation, yet its evaluation remains underdeveloped due t…
NVV-SuperBench: Beyond Words, Beyond Quality-Benchmarking Nonverbal Vocalizations in Speech Generation
Liumeng Xue, Weizhen Bian, Jiahao Pan +9
Nonverbal vocalizations (NVVs), such as laughing, sighing, and sobbing, are essential for human-like speech, yet standardized evaluation rarely jointly assesses whether systems gen…
FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection
Chengyou Wang, Hongfei Xue, Mingchen Shao +9
Recent advances in AudioLLMs have enabled spoken dialogue systems to move beyond turn-based interaction toward real-time full-duplex communication, where the agent must decide when…
OmniCodec: Low Frame Rate Universal Audio Codec with Semantic-Acoustic Disentanglement
Jingbin Hu, Haoyu Zhang, Dake Guo +10
Large Language Models (LLMs) have advanced audio generation through discrete representation learning. However, most existing neural codecs focus on speech and emphasize reconstruct…
Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning
Wenjie Tian, Mingchen Shao, Bingshen Mu +8
Audio-visual speech recognition (AVSR) is an extension of ASR that incorporates visual signals. Current AVSR approaches primarily focus on lip motion, largely overlooking rich cont…
EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs
Wenjie Tian, Zhixian Zhao, Jingbin Hu +4
The evolution of Omni-Modal Large Language Models~(Omni-LLMs) has revolutionized human--computer interaction, enabling unified audio-visual perception and speech response. However,…
VoiceSculptor: Your Voice, Designed By You
Jingbin Hu, Huakang Chen, Linhan Ma +19
Despite rapid progress in text-to-speech (TTS), open-source systems still lack truly instruction-following, fine-grained control over core speech attributes (e.g., pitch, speaking…