21 citations · 90 across the 17 of their papers we have counts for
14 papers · 1 filter
NExT-Chat: An LMM for Chat, Detection and Segmentation
Ao Zhang, Yuan Yao, Wei Ji +2
The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to…
Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching
Meng Chu, Zhedong Zheng, Wei Ji +2
Navigating drones through natural language commands remains challenging due to the dearth of accessible multi-modal datasets and the stringent precision requirements for aligning v…
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
Juncheng Li, Kaihang Pan, Zhiqi Ge +7
Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize.…
Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment
Shengqiong Wu, Hao Fei, Wei Ji +1
Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during tran…
Generating Visual Spatial Description via Holistic 3D Scene Understanding
Yu Zhao, Hao Fei, Wei Ji +4
Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. Existing VSD work merely models the 2D geometrical v…
VPGTrans: Transfer Visual Prompt Generator across LLMs
Ao Zhang, Hao Fei, Yuan Yao +4
While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a co…
Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
Juncheng Li, Minghe Gao, Longhui Wei +7
Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by l…
MRTNet: Multi-Resolution Temporal Network for Video Sentence Grounding
Wei Ji, Long Chen, Yinwei Wei +2
Given an untrimmed video and natural language query, video sentence grounding aims to localize the target temporal moment in the video. Existing methods mainly tackle this task by…
Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization
Yiyang Chen, Zhedong Zheng, Wei Ji +2
We investigate composed image retrieval with text feedback. Users gradually look for the target of interest by moving from coarse to fine-grained feedback. However, existing method…
Invariant Grounding for Video Question Answering
Yicong Li, Xiang Wang, Junbin Xiao +2
Video Question Answering (VideoQA) is the task of answering questions about a video. At its core is understanding the alignments between visual scenes in video and linguistic seman…
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
Yuan Yao, Qianyu Chen, Ao Zhang +4
Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mai…
3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective
Zhedong Zheng, Jiayin Zhu, Wei Ji +2
This research aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image. Compared with…
Fine-Grained Scene Graph Generation with Data Transfer
Ao Zhang, Yuan Yao, Qianyu Chen +4
Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for h…
Video Question Answering: Datasets, Algorithms and Challenges
Yaoyao Zhong, Junbin Xiao, Wei Ji +3
Video Question Answering (VideoQA) aims to answer natural language questions according to the given videos. It has earned increasing attention with recent research trends in joint…