7 papers · 1 filter
Stage-adaptive Token Selection for Efficient Omni-modal LLMs
Zijie Xin, Jie Yang, Ruixiang Zhao +4
The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens th…
OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding
Ruixiang Zhao, Jie Yang, Zijie Xin +4
Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emerging capability of omni-moda…
Fundus-R1: Training a Fundus-Reading MLLM on Public Data
Yuchuan Deng, Qijie Wei, Kaiheng Qian +8
Fundus imaging such as CFP, OCT and UWF is crucial for the early detection of retinal anomalies and diseases. Fundus image understanding, due to its knowledge-intensive nature, pos…
SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval
Ruixiang Zhao, Zhihao Xu, Bangxiang Lan +3
For video-text retrieval, the use of CLIP has been a de facto choice. Since CLIP provides only image and text encoders, this consensus has led to a biased paradigm that entirely ig…
Learning Partially-Decorrelated Common Spaces for Ad-hoc Video Search
Fan Hu, Zijie Xin, Xirong Li
Ad-hoc Video Search (AVS) involves using a textual query to search for multiple relevant videos in a large collection of unlabeled short videos. The main challenge of AVS is the vi…
Multi-Object Sketch Animation by Scene Decomposition and Motion Planning
Jingyu Liu, Zijie Xin, Yuhan Fu +3
Sketch animation, which brings static sketches to life by generating dynamic video sequences, has found widespread applications in GIF design, cartoon production, and daily enterta…
DAPL: Integration of Positive and Negative Descriptions in Text-Based Person Search
Yuchuan Deng, Zhanpeng Hu, Zijie Xin +2
Text-based person search (TBPS) aims to retrieve specific images of individuals from large datasets using textual descriptions. Existing TBPS methods focus primarily on identifying…