InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Abstract

Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-sourced at https://github.com/salesforce/LAVIS/tree/main/projects/instructblip.

arXiv commentpreprint

Cited by in corpus (17)

• Vision-Language Models for Medical Report Generation and Visual Question Answering: A Review
• UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility
• Evolutionary Optimization of Model Merging Recipes
• A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective
• Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
• IFShip: Interpretable Fine-grained Ship Classification with Domain Knowledge-Enhanced Vision-Language Models
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
• Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning
• Unsupervised Text Embedding Space Generation Using Generative Adversarial Networks for Text Synthesis
• Visual hallucination detection in large vision-language models via evidential conflict
• Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models
• GenCeption: Evaluate Vision LLMs with Unlabeled Unimodal Data
• TPA3D: Triplane Attention for Fast Text-to-3D Generation
• mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning
• Image-guided topic modeling for interpretable privacy classification
• MoralCLIP: Contrastive Alignment of Vision-and-Language Representations with Moral Foundations Theory