MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity

Abstract

Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of Vision Large Language Models (VLLMs). However, existing visual instruction tuning datasets include the following limitations: (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. There are four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering and Short Visual Question Answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments, we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at https://github.com/yuecao0119/MMInstruct.

arXiv comment18 pages, 8 figures, technical report

References in corpus (59)

• Learning Transferable Visual Models From Natural Language Supervision
• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Flamingo: a Visual Language Model for Few-Shot Learning
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• LAION-5B: An open large-scale dataset for training next generation image-text models
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• Gemini: A Family of Highly Capable Multimodal Models
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• PaLI: A Jointly-Scaled Multilingual Language-Image Model
• FigureQA: An Annotated Figure Dataset for Visual Reasoning
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• VideoChat: Chat-Centric Video Understanding
• Multimodal Few-Shot Learning with Frozen Language Models
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
• MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
• LISA: Reasoning Segmentation via Large Language Model
• Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark
• MMBench: Is Your Multi-modal Model an All-around Player?
• Valley: Video Assistant with Large Language model Enhanced abilitY
• Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs
• MIMIC-IT: Multi-Modal In-Context Instruction Tuning
• Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
• DVQA: Understanding Data Visualizations via Question Answering
• LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
• VL-BEiT: Generative Vision-Language Pretraining
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
• M${}^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
• Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning
• VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
• Evaluating Object Hallucination in Large Vision-Language Models
• The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• MapQA: A Dataset for Question Answering on Choropleth Maps
• InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
• MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer
• TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
• VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
• Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
• The All-Seeing Project V2: Towards General Relation Comprehension of the Open World
• Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
• UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
• Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
• InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions