EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment

Abstract

The furnishing of multi-modal large language models (MLLMs) has led to the emergence of numerous benchmark studies, particularly those evaluating their perception and understanding capabilities. Among these, understanding image-evoked emotions aims to enhance MLLMs' empathy, with significant applications such as human-machine interaction and advertising recommendations. However, current evaluations of this MLLM capability remain coarse-grained, and a systematic and comprehensive assessment is still lacking. To this end, we introduce EEmo-Bench, a novel benchmark dedicated to the analysis of the evoked emotions in images across diverse content categories. Our core contributions include: 1) Regarding the diversity of the evoked emotions, we adopt an emotion ranking strategy and employ the Valence-Arousal-Dominance (VAD) as emotional attributes for emotional assessment. In line with this methodology, 1,960 images are collected and manually annotated. 2) We design four tasks to evaluate MLLMs' ability to capture the evoked emotions by single images and their associated attributes: Perception, Ranking, Description, and Assessment. Additionally, image-pairwise analysis is introduced to investigate the model's proficiency in performing joint and comparative analysis. In total, we collect 6,773 question-answer pairs and perform a thorough assessment on 19 commonly-used MLLMs. The results indicate that while some proprietary and large-scale open-source MLLMs achieve promising overall performance, the analytical capabilities in certain evaluation dimensions remain suboptimal. Our EEmo-Bench paves the path for further research aimed at enhancing the comprehensive perceiving and understanding capabilities of MLLMs concerning image-evoked emotions, which is crucial for machine-centric emotion perception and understanding.

References in corpus (14)

• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Qwen2.5-VL Technical Report
• DeepSeek-VL: Towards Real-World Vision-Language Understanding
• LLaVA-OneVision: Easy Visual Task Transfer
• LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
• AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
• A Survey on Benchmarks of Multimodal Large Language Models
• Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMs
• UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark