579 citations · 3k across the 98 of their papers we have counts for
31 papers · 1 filter
Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation
Han Liu, Yinwei Wei, Fan Liu +3
Multimodal information (e.g., visual, acoustic, and textual) has been widely used to enhance representation learning for micro-video recommendation. For integrating multimodal info…
LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation
Leigang Qu, Shengqiong Wu, Hao Fei +2
In the text-to-image generation field, recent remarkable progress in Stable Diffusion makes it possible to generate rich kinds of novel photorealistic images. However, current mode…
Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling
Yu Zhao, Hao Fei, Yixin Cao +5
Video Semantic Role Labeling (VidSRL) aims to detect the salient events from given videos, by recognizing the predict-argument event structures and the interrelationships between e…
Redundancy-aware Transformer for Video Question Answering
Yicong Li, Xun Yang, An Zhang +3
This paper identifies two kinds of redundancy in the current VideoQA paradigm. Specifically, the current video encoders tend to holistically embed all video clues at different gran…
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
Juncheng Li, Kaihang Pan, Zhiqi Ge +7
Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize.…
Discovering Spatio-Temporal Rationales for Video Question Answering
Yicong Li, Junbin Xiao, Chun Feng +2
This paper strives to solve complex video question answering (VideoQA) which features long video containing multiple objects and events at different time. To tackle the challenge,…
Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment
Shengqiong Wu, Hao Fei, Wei Ji +1
Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during tran…
Generating Visual Spatial Description via Holistic 3D Scene Understanding
Yu Zhao, Hao Fei, Wei Ji +4
Visual spatial description (VSD) aims to generate texts that describe the spatial relations of the given objects within images. Existing VSD work merely models the 2D geometrical v…
VPGTrans: Transfer Visual Prompt Generator across LLMs
Ao Zhang, Hao Fei, Yuan Yao +4
While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a co…
Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
Juncheng Li, Minghe Gao, Longhui Wei +7
Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by l…
Contrastive Video Question Answering via Video Graph Transformer
Junbin Xiao, Pan Zhou, Angela Yao +4
We propose to perform video question answering (VideoQA) in a Contrastive manner via a Video Graph Transformer model (CoVGT). CoVGT's uniqueness and superiority are three-fold: 1)…
Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Grounding
Juncheng Li, Siliang Tang, Linchao Zhu +5
Temporal grounding is the task of locating a specific segment from an untrimmed video according to a query sentence. This task has achieved significant momentum in the computer vis…
MRTNet: Multi-Resolution Temporal Network for Video Sentence Grounding
Wei Ji, Long Chen, Yinwei Wei +2
Given an untrimmed video and natural language query, video sentence grounding aims to localize the target temporal moment in the video. Existing methods mainly tackle this task by…
Cognitive Accident Prediction in Driving Scenes: A Multimodality Benchmark
Jianwu Fang, Lei-Lei Li, Kuan Yang +3
Traffic accident prediction in driving videos aims to provide an early warning of the accident occurrence, and supports the decision making of safe driving systems. Previous works…
Visually Grounded Commonsense Knowledge Acquisition
Yuan Yao, Tianyu Yu, Ao Zhang +8
Large-scale commonsense knowledge bases empower a broad range of AI applications, where the automatic extraction of commonsense knowledge (CKE) is a fundamental and challenging pro…
Behavioral Intention Prediction in Driving Scenes: A Survey
Jianwu Fang, Fan Wang, Jianru Xue +1
In the driving scene, the road agents usually conduct frequent interactions and intention understanding of the surroundings. Ego-agent (each road agent itself) predicts what behavi…
Video Graph Transformer for Video Question Answering
Junbin Xiao, Pan Zhou, Tat-Seng Chua +1
This paper proposes a Video Graph Transformer (VGT) model for Video Quetion Answering (VideoQA). VGT's uniqueness are two-fold: 1) it designs a dynamic graph transformer module whi…
Towards Complex Document Understanding By Discrete Reasoning
Fengbin Zhu, Wenqiang Lei, Fuli Feng +3
Document Visual Question Answering (VQA) aims to understand visually-rich documents to answer questions in natural language, which is an emerging research topic for both Natural La…
Invariant Grounding for Video Question Answering
Yicong Li, Xiang Wang, Junbin Xiao +2
Video Question Answering (VideoQA) is the task of answering questions about a video. At its core is understanding the alignments between visual scenes in video and linguistic seman…
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
Yuan Yao, Qianyu Chen, Ao Zhang +4
Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mai…
3D Magic Mirror: Clothing Reconstruction from a Single Image via a Causal Perspective
Zhedong Zheng, Jiayin Zhu, Wei Ji +2
This research aims to study a self-supervised 3D clothing reconstruction method, which recovers the geometry shape and texture of human clothing from a single image. Compared with…
Fine-Grained Scene Graph Generation with Data Transfer
Ao Zhang, Yuan Yao, Qianyu Chen +4
Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for h…
Video Question Answering: Datasets, Algorithms and Challenges
Yaoyao Zhong, Junbin Xiao, Wei Ji +3
Video Question Answering (VideoQA) aims to answer natural language questions according to the given videos. It has earned increasing attention with recent research trends in joint…
Rethinking the Two-Stage Framework for Grounded Situation Recognition
Meng Wei, Long Chen, Wei Ji +2
Grounded Situation Recognition (GSR), i.e., recognizing the salient activity (or verb) category in an image (e.g., buying) and detecting all corresponding semantic roles (e.g., age…