924 citations · 2.7k across the 21 of their papers we have counts for
18 papers · 1 filter
DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models
Han Su, Meng Tian, Tianyu Huang +7
Vision-language models still struggle to understand 3D spatial relations across views and update spatial judgments as the observer moves. A common approach fuses 3D reconstruction…
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation
Ziyang Luo, Haoning Wu, Dongxu Li +3
Large multimodal models (LMMs) with advanced video analysis capabilities have recently garnered significant attention. However, most evaluations rely on traditional methods like mu…
Aria: An Open Multimodal Native Mixture-of-Experts Model
Dongxu Li, Yudong Liu, Haoning Wu +17
Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary mu…
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
Haoning Wu, Dongxu Li, Bei Chen +1
Large multimodal models (LMMs) are processing increasingly longer and richer inputs. Albeit the progress, few public benchmark is available to measure such development. To mitigate…
BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing
Dongxu Li, Junnan Li, Steven C. H. Hoi
Subject-driven text-to-image generation models create novel renditions of an input subject based on text prompts. Existing models suffer from lengthy fine-tuning and difficulties p…
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Wenliang Dai, Junnan Li, Dongxu Li +6
Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-lan…
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Junnan Li, Dongxu Li, Silvio Savarese +1
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficie…
From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models
Jiaxian Guo, Junnan Li, Dongxu Li +4
Large language models (LLMs) have demonstrated excellent zero-shot generalization to new language tasks. However, effective utilization of LLMs for zero-shot visual question-answer…
LAVIS: A Library for Language-Vision Intelligence
Dongxu Li, Junnan Li, Hung Le +3
We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. LAVIS aims to serve as a one-stop comprehensive library that brings recent a…
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Junnan Li, Dongxu Li, Caiming Xiong +1
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based t…
Align and Prompt: Video-and-Language Pre-training with Entity Prompts
Dongxu Li, Junnan Li, Hongdong Li +2
Video-and-language pre-training has shown promising improvements on various downstream tasks. Most previous methods capture cross-modal interactions with a transformer-based multim…
EDFace-Celeb-1M: Benchmarking Face Hallucination with a Million-scale Dataset
Kaihao Zhang, Dongxu Li, Wenhan Luo +4
Recent deep face hallucination methods show stunning performance in super-resolving severely degraded facial images, even surpassing human ability. However, these algorithms are ma…
ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring
Dongxu Li, Chenchen Xu, Kaihao Zhang +5
Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions. In order to utilize neighboring sharp patches, typical methods rely mainly…
Enhanced Spatio-Temporal Interaction Learning for Video Deraining: A Faster and Better Framework
Kaihao Zhang, Dongxu Li, Wenhan Luo +2
Video deraining is an important task in computer vision as the unwanted rain hampers the visibility of videos and deteriorates the robustness of most outdoor vision systems. Despit…
Dual Attention-in-Attention Model for Joint Rain Streak and Raindrop Removal
Kaihao Zhang, Dongxu Li, Wenhan Luo +1
Rain streaks and rain drops are two natural phenomena, which degrade image capture in different ways. Currently, most existing deep deraining networks take them as two distinct pro…
TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
Dongxu Li, Chenchen Xu, Xin Yu +4
Sign language translation (SLT) aims to interpret sign video sequences into text-based natural language sentences. Sign videos consist of continuous sequences of sign gestures with…
Transferring Cross-domain Knowledge for Video Sign Language Recognition
Dongxu Li, Xin Yu, Chenchen Xu +2
Word-level sign language recognition (WSLR) is a fundamental task in sign language interpretation. It requires models to recognize isolated sign words from videos. However, annotat…
Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison
Dongxu Li, Cristian Rodriguez Opazo, Xin Yu +1
Vision-based sign language recognition aims at helping deaf people to communicate with others. However, most existing sign language datasets are limited to a small number of words.…