4 papers · 1 filter
FATE: Frame-Level Audio-Visual Temporal Embedding
Kaisi Guan, Bingzi Zhang, Xihua Wang +4
When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations…
AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression
Yijing Chen, Wenhui Tan, Xiaoyi Yu +7
Multimodal Large Language Models have achieved remarkable progress in short-form audio-video understanding, yet long-form audio-video comprehension remains challenged by limited co…
Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts
Yuyue Wang, Xihua Wang, Xin Cheng +2
Audio generation has made significant progress, yet synthesizing unified audio where speech and sounds are naturally composited remains a challenge. Current methods either rely on…
MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
Wenhui Tan, Xiaoyi Yu, Jiaze Li +5
Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJ…