MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity

Abstract

Despite the effectiveness of vision-language supervised fine-tuning in enhancing the performance of Vision Large Language Models (VLLMs). However, existing visual instruction tuning datasets include the following limitations: (1) Instruction annotation quality: despite existing VLLMs exhibiting strong performance, instructions generated by those advanced VLLMs may still suffer from inaccuracies, such as hallucinations. (2) Instructions and image diversity: the limited range of instruction types and the lack of diversity in image data may impact the model's ability to generate diversified and closer to real-world scenarios outputs. To address these challenges, we construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. There are four instruction types: Judgement, Multiple-Choice, Long Visual Question Answering and Short Visual Question Answering. To construct MMInstruct, we propose an instruction generation data engine that leverages GPT-4V, GPT-3.5, and manual correction. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at 1/6 the cost of manual construction. Through extensive experiment validation and ablation experiments, we demonstrate that MMInstruct could significantly improve the performance of VLLMs, e.g., the model fine-tuning on MMInstruct achieves new state-of-the-art performance on 10 out of 12 benchmarks. The code and data shall be available at https://github.com/yuecao0119/MMInstruct.

18 pages, 8 figures, technical report

References in corpus (28)

• Learning Transferable Visual Models From Natural Language Supervision
• Training language models to follow instructions with human feedback
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• Multimodal Few-Shot Learning with Frozen Language Models
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
• Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs
• MIMIC-IT: Multi-Modal In-Context Instruction Tuning
• Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
• M${}^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning
• The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• MapQA: A Dataset for Question Answering on Choropleth Maps
• InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
• MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer
• VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
• Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks
• Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
• UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression
• InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions