MULTI: Multimodal Understanding Leaderboard with Text and Images
arXiv:2402.03173 · doi:10.1007/s11432-024-4602-x
Abstract
The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18,000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, We also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4,500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI.
24 pages, 19 figures, 10 tables. Details and access are available at: https://OpenDFM.github.io/MULTI-Benchmark/
References in corpus (14)
- LLaMA: Open and Efficient Foundation Language Models
- Gemini: A Family of Highly Capable Multimodal Models
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- Developing ChemDFM as a large language foundation model for chemistry
- InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition
- InternLM2 Technical Report
- MiniCPM-V: A GPT-4V Level MLLM on Your Phone
- SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs
- ChemDFM-X: Towards Large Multimodal Model for Chemistry
- TouchStone: Evaluating Vision-Language Models by Language Models
- Depression Diagnosis Dialogue Simulation: Self-improving Psychiatrist with Tertiary Memory
- SEED-Bench-2: Benchmarking Multimodal Large Language Models
- CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark