Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework

Abstract

Affect recognition based on subjects' facial expressions has been a topic of major research in the attempt to generate machines that can understand the way subjects feel, act and react. In the past, due to the unavailability of large amounts of data captured in real-life situations, research has mainly focused on controlled environments. However, recently, social media and platforms have been widely used. Moreover, deep learning has emerged as a means to solve visual analysis and recognition problems. This paper exploits these advances and presents significant contributions for affect analysis and recognition in-the-wild. Affect analysis and recognition can be seen as a dual knowledge generation problem, involving: i) creation of new, large and rich in-the-wild databases and ii) design and training of novel deep neural architectures that are able to analyse affect over these databases and to successfully generalise their performance on other datasets. The paper focuses on large in-the-wild databases, i.e., Aff-Wild and Aff-Wild2 and presents the design of two classes of deep neural networks trained with these databases. The first class refers to uni-task affect recognition, focusing on prediction of the valence and arousal dimensional variables. The second class refers to estimation of all main behavior tasks, i.e. valence-arousal prediction; categorical emotion classification in seven basic facial expressions; facial Action Unit detection. A novel multi-task and holistic framework is presented which is able to jointly learn and effectively generalize and perform affect recognition over all existing in-the-wild databases. Large experimental studies illustrate the achieved performance improvement over the existing state-of-the-art in affect recognition.

References in corpus (2)

• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• EmotioNet Challenge: Recognition of facial expressions of emotion in the wild

Cited by in corpus (19)

• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• From the Lab to the Wild: Affect Modeling via Privileged Information
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
• Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Feature Pyramid Network for Multi-task Affective Analysis
• Multi-modal Affect Analysis using standardized data within subjects in the Wild
• Spatial and Temporal Networks for Facial Expression Recognition in the Wild Videos
• A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis
• Action Unit Detection with Joint Adaptive Attention and Graph Relation
• Action Units Recognition Using Improved Pairwise Deep Architecture
• Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion
• Technical Report for Valence-Arousal Estimation in ABAW2 Challenge
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Bayesian Convolutional Neural Networks for Seven Basic Facial Expression Classifications
• Causal affect prediction model using a facial image sequence
• Seven Basic Expression Recognition Using ResNet-18