12 papers
MVWeaver: A Hierarchical Music Video Generation Agent with a Learned Song-to-Visual Bridge
Sifei Li, Minyan Luo, Xu Li +7
Music videos are an important form of audiovisual expression in contemporary culture. They translate and extend the expressive content of songs through deliberate visual design. Ex…
Closing the Verification Loop: Self-Check Captioning for Long-Paragraph Detailed Audio Captioning
Fengji Ma, Yan Rong, Xu Li +3
Long-paragraph detailed audio captioning, which requires dense and transcript-faithful descriptions of fine-grained audio content, remains unsolved for current audio-visual multimo…
ACE-Cap: Active Evidence Acquisition via Agentic Co-Evolution for Long-Paragraph Fine-Grained Audio Captioning
Fengji Ma, Yan Rong, Xu Li +3
Long-paragraph fine-grained audio captioning requires models to recover diverse acoustic facts while avoiding omissions and unsupported details. However, prevailing captioners rema…
SingDance: Compositional Zero-Shot Singing-and-Dancing Video Generation with Role-Aware Audio Conditioning
Tao Feng, Xu Li, Xiangyang Luo +4
Generating personalized dance videos from a reference image, text prompt, and audio track requires music-conditioned body motion. Singing-and-dancing adds a second requirement: the…
SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation
Yunrui Cai, Xu Li, Yucheng Zhou +8
Text-conditioned general audio generation is moving beyond isolated speech, music, and sound-effect synthesis toward a single model that can compose them into controllable, coheren…
AudioMap: Cloze-and-Choice Reinforcement Learning for Time-Aware Dense Audio Captioning
Yan Rong, Fengji Ma, Xu Li +3
Time-aware dense audio captioning (TDAC) aims to generate multiple fine-grained attributes (dense) of the audio with precise time boundaries (time-aware). Existing methods struggle…
Towards Unified Song Generation and Singing Voice Conversion with Accompaniment Co-Generation
Ziyu Zhang, Chunyu Qiang, Xiaopeng Wang +10
While song generation and singing voice conversion (SVC) have evolved significantly, they have long been developed isolated: the former lacks zero-shot speaker cloning, while the l…
SegTune: Structured and Fine-Grained Control for Song Generation
Yuejiao Wang, Zihao Ji, Pengfei Cai +6
Recent advances in neural song generation have enabled high-quality synthesis from lyrics and global textual prompts. However, most systems fail to model temporally varying attribu…
UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions
Chunyu Qiang, Xiaopeng Wang, Kang Yin +11
Generative audio modeling has largely been fragmented into specialized tasks, text-to-speech (TTS), text-to-music (TTM), and text-to-audio (TTA), each operating under heterogeneous…
SegTune: Structured and Fine-Grained Control for Song Generation
Pengfei Cai, Joanna Wang, Haorui Zheng +6
Recent advancements in song generation have shown promising results in generating songs from lyrics and/or global text prompts. However, most existing systems lack the ability to m…
Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
Jun Wang, Xijuan Zeng, Chunyu Qiang +19
We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce m…
Towards Flow-Matching-based TTS without Classifier-Free Guidance
Yuzhe Liang, Wenzhe Liu, Chunyu Qiang +7
Flow matching has demonstrated strong generative capabilities and has become a core component in modern Text-to-Speech (TTS) systems. To ensure high-quality speech synthesis, Class…