Multi-modal Affect Analysis using standardized data within subjects in the Wild

Abstract

Human affective recognition is an important factor in human-computer interaction. However, the method development with in-the-wild data is not yet accurate enough for practical usage. In this paper, we introduce the affective recognition method focusing on facial expression (EXP) and valence-arousal calculation that was submitted to the Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest. When annotating facial expressions from a video, we thought that it would be judged not only from the features common to all people, but also from the relative changes in the time series of individuals. Therefore, after learning the common features for each frame, we constructed a facial expression estimation model and valence-arousal model using time-series data after combining the common features and the standardized features for each video. Furthermore, the above features were learned using multi-modal data such as image features, AU, Head pose, and Gaze. In the validation set, our model achieved a facial expression score of 0.546. These verification results reveal that our proposed framework can improve estimation accuracy and robustness effectively.

arXiv comment6 pages, 5 figures

References in corpus (9)

• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose
• Affect Expression Behaviour Analysis in the Wild using Spatio-Channel Attention and Complementary Context Information
• Action Units Recognition by Pairwise Deep Architecture
• An Audio-Video Deep and Transfer Learning Framework for Multimodal Emotion Recognition in the wild

Cited by in corpus (2)

• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations