352 citations · 690 across the 42 of their papers we have counts for
45 papers
Benchmarking Foundation and Large Language Models for Few-Shot Medical Image Segmentation
Jinghong Liu, Yuchuan Deng, Fanping Liu +2
Few-shot medical image segmentation (FS-MIS) aims to segment novel regions of interest (ROIs) from a few annotated support examples. Despite rapid progress, existing FS-MIS solutio…
Stage-adaptive Token Selection for Efficient Omni-modal LLMs
Zijie Xin, Jie Yang, Ruixiang Zhao +4
The unified audio-visual understanding capability of omni-modal large language models (om-LLMs) comes at the high cost of processing tens of thousands of visual and audio tokens th…
OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding
Ruixiang Zhao, Jie Yang, Zijie Xin +4
Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emerging capability of omni-moda…
Fundus-R1: Training a Fundus-Reading MLLM on Public Data
Yuchuan Deng, Qijie Wei, Kaiheng Qian +8
Fundus imaging such as CFP, OCT and UWF is crucial for the early detection of retinal anomalies and diseases. Fundus image understanding, due to its knowledge-intensive nature, pos…
Early Intervention for VFM-based Multimodal Medical Image Classification
Qijie Wei, Hailan Lin, Xirong Li
Current methods for multimodal medical image classification (M3IC) face two major challenges. First, the prevailing "fusion after unimodal image embedding" paradigm cannot fully ex…
SAVE: Speech-Aware Video Representation Learning for Video-Text Retrieval
Ruixiang Zhao, Zhihao Xu, Bangxiang Lan +3
For video-text retrieval, the use of CLIP has been a de facto choice. Since CLIP provides only image and text encoders, this consensus has led to a biased paradigm that entirely ig…
Cross-modal Fundus Image Registration under Large FoV Disparity
Hongyang Li, Junyi Tao, Qijie Wei +4
Previous work on cross-modal fundus image registration (CMFIR) assumes small cross-modal Field-of-View (FoV) disparity. By contrast, this paper is targeted at a more challenging sc…
AEFS: Adaptive Early Feature Selection for Deep Recommender Systems
Fan Hu, Gaofeng Lu, Jun Chen +3
Feature selection has emerged as a crucial technique in refining recommender systems. Recent advancements leveraging Automated Machine Learning (AutoML) has drawn significant atten…
Hybrid-Tower: Fine-grained Pseudo-query Interaction and Generation for Text-to-Video Retrieval
Bangxiang Lan, Ruobing Xie, Ruixiang Zhao +4
The Text-to-Video Retrieval (T2VR) task aims to retrieve unlabeled videos by textual queries with the same semantic meanings. Recent CLIP-based approaches have explored two framewo…
Learning Partially-Decorrelated Common Spaces for Ad-hoc Video Search
Fan Hu, Zijie Xin, Xirong Li
Ad-hoc Video Search (AVS) involves using a textual query to search for multiple relevant videos in a large collection of unlabeled short videos. The main challenge of AVS is the vi…
FunBench: Benchmarking Fundus Reading Skills of MLLMs
Qijie Wei, Kaiheng Qian, Xirong Li
Multimodal Large Language Models (MLLMs) have shown significant potential in medical image analysis. However, their capabilities in interpreting fundus images, a critical skill for…
Multi-Object Sketch Animation by Scene Decomposition and Motion Planning
Jingyu Liu, Zijie Xin, Yuhan Fu +3
Sketch animation, which brings static sketches to life by generating dynamic video sequences, has found widespread applications in GIF design, cartoon production, and daily enterta…
Convolutional Prompting for Broad-Domain Retinal Vessel Segmentation
Qijie Wei, Weihong Yu, Xirong Li
Previous research on retinal vessel segmentation is targeted at a specific image domain, mostly color fundus photography (CFP). In this paper we make a brave attempt to attack a mo…
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
Yuhan Fu, Ruobing Xie, Xingwu Sun +2
Multimodal Large Language Models (MLLMs) are known to hallucinate, which limits their practical applications. Recent works have attempted to apply Direct Preference Optimization (D…
Beyond Coarse-Grained Matching in Video-Text Retrieval
Aozhu Chen, Hazel Doughty, Xirong Li +1
Video-text retrieval has seen significant advancements, yet the ability of models to discern subtle differences in captions still requires verification. In this paper, we introduce…
Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructions
Yuhan Fu, Ruobing Xie, Jiazhen Liu +4
Hallucinations in multimodal large language models (MLLMs) hinder their practical applications. To address this, we propose a Magnifier Prompt (MagPrompt), a simple yet effective m…
Music Grounding by Short Video
Zijie Xin, Minquan Wang, Jingyu Liu +4
Adding proper background music helps complete a short video to be shared. Previous work tackles the task by video-to-music retrieval (V2MR), aiming to find the most suitable music…
D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matching
Jingyu Liu, Minquan Wang, Ye Ma +5
Videos showcasing specific products are increasingly important for E-commerce. Key moments naturally exist as the first appearance of a specific product, presentation of its distin…
ASR-enhanced Multimodal Representation Learning for Cross-Domain Product Retrieval
Ruixiang Zhao, Jian Jia, Yan Li +5
E-commerce is increasingly multimedia-enriched, with products exhibited in a broad-domain manner as images, short videos, or live stream promotions. A unified and vectorized cross-…
PhD: A ChatGPT-Prompted Visual hallucination Evaluation Dataset
Jiazhen Liu, Yuhan Fu, Ruobing Xie +5
Multimodal Large Language Models (MLLMs) hallucinate, resulting in an emerging topic of visual hallucination evaluation (VHE). This paper contributes a ChatGPT-Prompted visual hall…
CLIPRerank: An Extremely Simple Method for Improving Ad-hoc Video Search
Aozhu Chen, Fangming Zhou, Ziyuan Wang +1
Ad-hoc Video Search (AVS) enables users to search for unlabeled video content using on-the-fly textual queries. Current deep learning-based models for AVS are trained to optimize h…
TeachCLIP: Multi-Grained Teaching for Efficient Text-to-Video Retrieval
Kaibin Tian, Ruixiang Zhao, Hu Hu +4
For text-to-video retrieval (T2VR), which aims to retrieve unlabeled videos by ad-hoc textual queries, CLIP-based methods are dominating. Compared to CLIP4Clip which is efficient a…
Supervised Domain Adaptation for Recognizing Retinal Diseases from Wide-Field Fundus Images
Qijie Wei, Jingyuan Yang, Bo Wang +9
This paper addresses the emerging task of recognizing multiple retinal diseases from wide-field (WF) and ultra-wide-field (UWF) fundus images. For an effective use of existing larg…
ChinaOpen: A Dataset for Open-world Multimodal Learning
Aozhu Chen, Ziyuan Wang, Chengbo Dong +5
This paper introduces ChinaOpen, a dataset sourced from Bilibili, a popular Chinese video-sharing website, for open-world multimodal learning. While the state-of-the-art multimodal…