5 papers
Crab: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation
Dongnuan Cai, Henghui Du, Chang Zhou +5
Developing Audio-Visual Large Language Models (AV-LLMs) for unified scene understanding is pivotal in multimodal intelligence. While instruction tuning enables pre-trained models w…
APPO: Attention-guided Perception Policy Optimization for Video Reasoning
Henghui Du, Chang Zhou, Xi Chen +1
Complex video reasoning, actually, relies excessively on fine-grained perception rather than on expert (e.g., Ph.D, Science)-level reasoning. Through extensive empirical observatio…
Video Detective: Seek Critical Clues Recurrently to Answer Question from Long Videos
Henghui Du, Chunjie Zhang, Xi Chen +2
Long Video Question-Answering (LVQA) presents a significant challenge for Multi-modal Large Language Models (MLLMs) due to immense context and overloaded information, which could a…
Crab: A Unified Audio-Visual Scene Understanding Model with Explicit Cooperation
Henghui Du, Guangyao Li, Chang Zhou +3
In recent years, numerous tasks have been proposed to encourage model to develop specified capability in understanding audio-visual scene, primarily categorized into temporal local…
On-the-fly Modulation for Balanced Multimodal Learning
Yake Wei, Di Hu, Henghui Du +1
Multimodal learning is expected to boost model performance by integrating information from different modalities. However, its potential is not fully exploited because the widely-us…