ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild

Abstract

Humans are arguably innately prepared to comprehend others' emotional expressions from subtle body movements. If robots or computers can be empowered with this capability, a number of robotic applications become possible. Automatically recognizing human bodily expression in unconstrained situations, however, is daunting given the incomplete understanding of the relationship between emotional expressions and body movements. The current research, as a multidisciplinary effort among computer and information sciences, psychology, and statistics, proposes a scalable and reliable crowdsourcing approach for collecting in-the-wild perceived emotion data for computers to learn to recognize body languages of humans. To accomplish this task, a large and growing annotated dataset with 9,876 video clips of body movements and 13,239 human characters, named BoLD (Body Language Dataset), has been created. Comprehensive statistical analysis of the dataset revealed many interesting insights. A system to model the emotional expressions based on bodily movements, named ARBEE (Automated Recognition of Bodily Expression of Emotion), has also been developed and evaluated. Our analysis shows the effectiveness of Laban Movement Analysis (LMA) features in characterizing arousal, and our experiments using LMA features further demonstrate computability of bodily expression. We report and compare results of several other baseline methods which were developed for action recognition based on two different modalities, body skeleton, and raw image. The dataset and findings presented in this work will likely serve as a launchpad for future discoveries in body language understanding that will enable future robots to interact and collaborate more effectively with humans.

References in corpus (6)

• Semi-Supervised Classification with Graph Convolutional Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Simple Online and Realtime Tracking
• YFCC100M: The New Data in Multimedia Research
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

Cited by in corpus (12)

• Benchmarking Micro-action Recognition: Dataset, Methods, and Applications
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition
• Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild
• EmoStim: A Database of Emotional Film Clips with Discrete and Componential Assessment
• MoEmo Vision Transformer: Integrating Cross-Attention and Movement Vectors in 3D Pose Estimation for HRI Emotion Detection
• Incorporating simulated spatial context information improves the effectiveness of contrastive learning models
• iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild
• Simulation of Human and Artificial Emotion (SHArE)
• MEBOW: Monocular Estimation of Body Orientation In the Wild
• Design of a Human-Assistance Robot System with Contextual Action Recognition