11 papers
UniAE-MoE: A Unified Audio Encoder via Mixture of Experts
Shengbo Cai, Zhisheng Zhang, Zichao Nie +3
Large Audio Language Models (LALMs) rely on effective audio encoders for multi-task performance. We introduce UniAE-MoE, a unified audio encoder designed to model cross-domain audi…
Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs
Jing Peng, Zichao Nie, Zhisheng Zhang +2
Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated.…
LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training
Shun Lei, Huaicheng Zhang, Dapeng Wu +8
Full-length song generation must preserve coherence and musicality, render detailed vocal and accompaniment acoustics, and follow lyrics and prompts. Existing language model-based…
Self-Guidance: Enhancing Neural Codecs via Decoder Manifold Alignment
Xiang Li, Yixuan Zhou, Jingran Xie +2
Neural speech codecs based on Vector-Quantized VAEs (VQ-VAEs) are core audio tokenizers for speech LLMs, yet their reconstruction fidelity is bottlenecked by quantization error. Mo…
WAKE: Watermarking Audio with Key Enrichment
Yaoxun Xu, Jianwei Yu, Hangting Chen +5
As deep learning advances in audio generation, challenges in audio security and copyright protection highlight the need for robust audio watermarking. Recent neural network-based m…
LeVo: High-Quality Song Generation with Multi-Preference Alignment
Shun Lei, Yaoxun Xu, Zhiwei Lin +10
Recent advances in large language models (LLMs) and audio language models have significantly improved music generation, particularly in lyrics-to-song generation. However, existing…
VoiceMark: Zero-Shot Voice Cloning-Resistant Watermarking Approach Leveraging Speaker-Specific Latents
Haiyun Li, Zhiyong Wu, Xiaofeng Xie +3
Voice cloning (VC)-resistant watermarking is an emerging technique for tracing and preventing unauthorized cloning. Existing methods effectively trace traditional VC models by trai…
Comparing Discrete and Continuous Space LLMs for Speech Recognition
Yaoxun Xu, Shi-Xiong Zhang, Jianwei Yu +2
This paper investigates discrete and continuous speech representations in Large Language Model (LLM)-based Automatic Speech Recognition (ASR), organizing them by feature continuity…
HydraFormer: One Encoder For All Subsampling Rates
Yaoxun Xu, Xingchen Song, Zhiyong Wu +3
In automatic speech recognition, subsampling is essential for tackling diverse scenarios. However, the inadequacy of a single subsampling rate to address various real-world situati…
Consistent and Relevant: Rethink the Query Embedding in General Sound Separation
Yuanyuan Wang, Hangting Chen, Dongchao Yang +4
The query-based audio separation usually employs specific queries to extract target sources from a mixture of audio signals. Currently, most query-based separation models need addi…
SECap: Speech Emotion Captioning with Large Language Model
Yaoxun Xu, Hangting Chen, Jianwei Yu +6
Speech emotions are crucial in human communication and are extensively used in fields like speech synthesis and natural language understanding. Most prior studies, such as speech e…