Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion

Abstract

The emergence of artificial emotional intelligence technology is revolutionizing the fields of computers and robotics, allowing for a new level of communication and understanding of human behavior that was once thought impossible. While recent advancements in deep learning have transformed the field of computer vision, automated understanding of evoked or expressed emotions in visual media remains in its infancy. This foundering stems from the absence of a universally accepted definition of "emotion", coupled with the inherently subjective nature of emotions and their intricate nuances. In this article, we provide a comprehensive, multidisciplinary overview of the field of emotion analysis in visual media, drawing on insights from psychology, engineering, and the arts. We begin by exploring the psychological foundations of emotion and the computational principles that underpin the understanding of emotions from images and videos. We then review the latest research and systems within the field, accentuating the most promising approaches. We also discuss the current technological challenges and limitations of emotion analysis, underscoring the necessity for continued investigation and innovation. We contend that this represents a "Holy Grail" research problem in computing and delineate pivotal directions for future inquiry. Finally, we examine the ethical ramifications of emotion-understanding technologies and contemplate their potential societal impacts. Overall, this article endeavors to equip readers with a deeper understanding of the domain of emotion analysis in visual media and to inspire further research and development in this captivating and rapidly evolving field.

Proceedings of the IEEE 2023

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Kinetics Human Action Video Dataset
• EEG based Emotion Recognition: A Tutorial and Review
• Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks
• Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
• DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks
• Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies
• A Survey on Aspect-Based Sentiment Classification
• Visual Sentiment Prediction with Deep Convolutional Neural Networks
• Stimuli-Aware Visual Emotion Analysis
• SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network
• Latent Multi-group Membership Graph Model
• Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data
• TNTC: two-stream network with transformer-based complementarity for gait-based emotion recognition