9 papers
Visual Representation Matters: Exploiting Temporal Differences in Video-to-Audio Generation
Zehua Chen, Junyou Wang, Yuxuan Jiang +5
Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existin…
Vidu S1: A Real-Time Interactive Video Generation Model
Jintao Zhang, Kai Jiang, Jintao Chen +24
We introduce Vidu S1, a real-time interactive video generation model supporting voice control of digital characters. Users can control video generation content at any moment throug…
AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization
Tianhong Zhou, Mingyang Han, Boyu Li +8
Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for feature extraction models ex…
FreeSonic: Training-Free Temporal-Aware Decoupled Attention for Precise Audio Editing
Yuxuan Jiang, Mingyang Han, Yusheng Dai +12
Text-to-audio (TTA) generation has made significant strides, yet achieving precise and consistent audio editing remains a major challenge. However, existing methods struggle to bal…
Omni2Sound: Towards Unified Video-Text-to-Audio Generation
Yusheng Dai, Zehua Chen, Yuxuan Jiang +4
Training a unified model integrating video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) generation offers significant application flexibility, yet face…
ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling
Yuxuan Jiang, Zehua Chen, Zeqian Ju +3
Text-to-audio (TTA) generation with fine-grained control signals, e.g., precise timing control or intelligible speech content, has been explored in recent works. However, constrain…