4 papers
AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint Control
Xinyue Guo, Xiaoran Yang, Lipan Zhang +3
Sound effect editing-modifying audio by adding, removing, or replacing elements-remains constrained by existing approaches that rely solely on low-level signal processing or coarse…
MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimization
Jianxuan Yang, Xiaoran Yang, Lipan Zhang +3
Current video-to-audio (V2A) methods struggle in complex multi-event scenarios (video scenarios involving multiple sound sources, sound events, or transitions) due to two critical…
MSGCoOp: Multiple Semantic-Guided Context Optimization for Few-Shot Learning
Zhaolong Wang, Tongfeng Sun, Mingzheng Du +1
Vision-language pre-trained models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, and prompt learning has emerged as an efficient alternative to full fi…
Unlocking Pretrained LLMs for Motion-Related Multimodal Generation: A Fine-Tuning Approach to Unify Diffusion and Next-Token Prediction
Shinichi Tanaka, Zhao Wang, Yoichi Kato +1
In this paper, we propose a unified framework that leverages a single pretrained LLM for Motion-related Multimodal Generation, referred to as MoMug. MoMug integrates diffusion-base…