MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
arXiv:2304.10592
Abstract
The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.
arXiv commentProject Website: https://minigpt-4.github.io/; Code, Pretrained Model, and Dataset: https://github.com/Vision-CAIR/MiniGPT-4; Deyao Zhu and Jun Chen contributed equally to this work
Cited by in corpus (28)
- Unifying Large Language Models and Knowledge Graphs: A Roadmap
- Large language models in medicine: the potentials and pitfalls
- Taking the Next Step with Generative Artificial Intelligence: The Transformative Role of Multimodal Large Language Models in Science Education
- OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models
- Woodpecker: Hallucination Correction for Multimodal Large Language Models
- GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text
- Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective
- Generative AI and Process Systems Engineering: The Next Frontier
- Deepfake: Definitions, Performance Metrics and Standards, Datasets and Benchmarks, and a Meta-Review
- A Survey on Occupancy Perception for Autonomous Driving: The Information Fusion Perspective
- From Screens to Scenes: A Survey of Embodied AI in Healthcare
- Several categories of Large Language Models (LLMs): A Short Survey
- Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
- Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
- FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries
- Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks
- Foundation Models and Transformers for Anomaly Detection: A Survey
- RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models
- Power-LLaVA: Large Language and Vision Assistant for Power Transmission Line Inspection
- Automatic Medical Report Generation: Methods and Applications
- GPTSee: Enhancing Moment Retrieval and Highlight Detection via Description-Based Similarity Features
- IFShip: Interpretable Fine-grained Ship Classification with Domain Knowledge-Enhanced Vision-Language Models
- Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM
- MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity
- HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs
- ROSGPT_Vision: Commanding Robots Using Only Language Models' Prompts
- Reply with Sticker: New Dataset and Model for Sticker Retrieval
- CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation