409 citations · 677 across the 54 of their papers we have counts for
24 papers · 1 filter
REZE: Recognition-Based Zero-Shot Extraction for Video Temporal Grounding
Boyang Li, Chenhui Gou, Jianfei Cai
Video temporal grounding (VTG) refers to the task of identifying the time interval in a video that corresponds to a given natural-language query. A common zero-shot strategy asks a…
Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data
Haoxin Li, Boyang Li
Paired image-text data with subtle variations in-between (e.g., people holding surfboards vs. people holding shovels) hold the promise of producing Vision-Language Models with prop…
Learning to Animate Images from A Few Videos to Portray Delicate Human Actions
Haoxin Li, Yingchen Yu, Qilong Wu +3
Despite recent progress, video generative models still struggle to animate static images into videos that portray delicate human actions, particularly when handling uncommon or nov…
SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation
Wenyu Zhang, Wei En Ng, Lixin Ma +5
Current vision-language models may grasp basic spatial cues and simple directions (e.g. left, right, front, back), but struggle with the multi-dimensional spatial reasoning necessa…
The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding
Jiayun Luo, Mir Rayat Imtiaz Hossain, Pritam Sarkar +2
Vision-Language Models (VLMs) have achieved strong performance on implicit and explicit visual grounding and related tasks. However, such abilities are generally tested on simple,…
Black Swan: Abductive and Defeasible Video Reasoning in Unpredictable Events
Aditya Chinchure, Sahithya Ravi, Raymond Ng +3
The commonsense reasoning capabilities of vision-language models (VLMs), especially in abductive reasoning and defeasible reasoning, remain poorly understood. Most benchmarks focus…
Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Grounding
Zilin Du, Haoxin Li, Jianfei Yu +1
Visual grounding aims to localize the image regions based on a textual query. Given the difficulty of large-scale data curation, we investigate how to effectively learn visual grou…
Concept-skill Transferability-based Data Selection for Large Vision-Language Models
Jaewoo Lee, Boyang Li, Sung Ju Hwang
Instruction tuning, or supervised finetuning on extensive task-specific data, is necessary for Large Vision-Language Models (LVLMs) to generalize well across a broad range of visio…
What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases
Anthony Meng Huat Tiong, Junqi Zhao, Boyang Li +3
Vision-language (VL) models, pretrained on colossal image-text datasets, have attained broad VL competence that is difficult to evaluate. A common belief is that a small number of…
Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
Jiayun Luo, Siddhesh Khandelwal, Leonid Sigal +1
From image-text pairs, large-scale vision-language models (VLMs) learn to implicitly associate image regions with words, which prove effective for tasks like visual question answer…
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Wenliang Dai, Junnan Li, Dongxu Li +6
Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-lan…
From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models
Jiaxian Guo, Junnan Li, Dongxu Li +4
Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answer…
Mitigating and Evaluating Static Bias of Action Representations in the Background and the Foreground
Haoxin Li, Yuan Liu, Hanwang Zhang +1
In video action recognition, shortcut static features can interfere with the learning of motion features, resulting in poor out-of-distribution (OOD) generalization. The video back…
A Survey of Computer Vision Technologies In Urban and Controlled-environment Agriculture
Jiayun Luo, Boyang Li, Cyril Leung
In the evolution of agriculture to its next stage, Agriculture 5.0, artificial intelligence will play a central role. Controlled-environment agriculture, or CEA, is a special form…
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training
Anthony Meng Huat Tiong, Junnan Li, Boyang Li +2
Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular…
Efficient Self-supervised Vision Pretraining with Local Masked Reconstruction
Jun Chen, Ming Hu, Boyang Li +1
Self-supervised learning for computer vision has achieved tremendous progress and improved many downstream vision tasks such as image classification, semantic segmentation, and obj…
Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding
Yidan Sun, Qin Chao, Yangfeng Ji +1
Despite recent advances of AI, story understanding remains an open and under-investigated problem. We collect, preprocess, and publicly release a video-language story dataset, Syno…
Improving Tail-Class Representation with Centroid Contrastive Learning
Anthony Meng Huat Tiong, Junnan Li, Guosheng Lin +3
In vision domain, large-scale natural datasets typically exhibit long-tailed distribution which has large class imbalance between head and tail classes. This distribution poses dif…
Noise-resistant Deep Metric Learning with Ranking-based Instance Selection
Chang Liu, Han Yu, Boyang Li +6
The existence of noisy labels in real-world data negatively impacts the performance of deep learning models. Although much research effort has been devoted to improving robustness…
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Jun Chen, Han Guo, Kai Yi +2
The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning mo…
Exploring Long Tail Visual Relationship Recognition with Large Vocabulary
Sherif Abdelkarim, Aniket Agarwal, Panos Achlioptas +5
Several approaches have been proposed in recent literature to alleviate the long-tail problem, mainly in object classification tasks. In this paper, we make the first large-scale s…
A Neural Multi-sequence Alignment TeCHnique (NeuMATCH)
Pelin Dogan, Boyang Li, Leonid Sigal +1
The alignment of heterogeneous sequential data (video to text) is an important and challenging problem. Standard techniques for this task, including Dynamic Time Warping (DTW) and…
Joint Event Detection and Description in Continuous Video Streams
Huijuan Xu, Boyang Li, Vasili Ramanishka +2
Dense video captioning is a fine-grained video understanding task that involves two sub-problems: localizing distinct events in a long video stream, and generating captions for the…
Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization
Baohan Xu, Yanwei Fu, Yu-Gang Jiang +2
Emotion is a key element in user-generated videos. However, it is difficult to understand emotions conveyed in such videos due to the complex and unstructured nature of user-genera…