Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace

Abstract

Affective computing has been largely limited in terms of available data resources. The need to collect and annotate diverse in-the-wild datasets has become apparent with the rise of deep learning models, as the default approach to address any computer vision task. Some in-the-wild databases have been recently proposed. However: i) their size is small, ii) they are not audiovisual, iii) only a small part is manually annotated, iv) they contain a small number of subjects, or v) they are not annotated for all main behavior tasks (valence-arousal estimation, action unit detection and basic expression classification). To address these, we substantially extend the largest available in-the-wild database (Aff-Wild) to study continuous emotions such as valence and arousal. Furthermore, we annotate parts of the database with basic expressions and action units. As a consequence, for the first time, this allows the joint study of all three types of behavior states. We call this database Aff-Wild2. We conduct extensive experiments with CNN and CNN-RNN architectures that use visual and audio modalities; these networks are trained on Aff-Wild2 and their performance is then evaluated on 10 publicly available emotion databases. We show that the networks achieve state-of-the-art performance for the emotion recognition tasks. Additionally, we adapt the ArcFace loss function in the emotion recognition context and use it for training two new networks on Aff-Wild2 and then re-train them in a variety of diverse expression recognition databases. The networks are shown to improve the existing state-of-the-art. The database, emotion recognition models and source code are available at http://ibug.doc.ic.ac.uk/resources/aff-wild2.

oral presentation in BMVC 2019

References in corpus (4)

• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
• Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition

Cited by in corpus (46)

• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• Multitask Emotion Recognition with Incomplete Labels
• Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique
• Feature Pyramid Network for Multi-task Affective Analysis
• Emotion Recognition for In-the-wild Videos
• Affect Expression Behaviour Analysis in the Wild using Spatio-Channel Attention and Complementary Context Information
• $M^3$T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild
• Action Units Recognition by Pairwise Deep Architecture
• Multi-modal Affect Analysis using standardized data within subjects in the Wild
• Spatial and Temporal Networks for Facial Expression Recognition in the Wild Videos
• Robust Training of Social Media Image Classification Models for Rapid Disaster Response
• Technical Report for Valence-Arousal Estimation on Affwild2 Dataset
• A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis
• Adversarial-based neural networks for affect estimations in the wild
• Multi-label Relation Modeling in Facial Action Units Detection
• A Multi-term and Multi-task Analyzing Framework for Affective Analysis in-the-wild
• Action Units Recognition Using Improved Pairwise Deep Architecture
• Aff-Wild Database and AffWildNet
• Multi-Label Class Balancing Algorithm for Action Unit Detection
• MAFER: a Multi-resolution Approach to Facial Expression Recognition
• Lossless Attention in Convolutional Networks for Facial Expression Recognition in the Wild
• Action Unit Detection with Joint Adaptive Attention and Graph Relation
• An Audio-Video Deep and Transfer Learning Framework for Multimodal Emotion Recognition in the wild
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Technical Report for Valence-Arousal Estimation in ABAW2 Challenge
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model
• Multi-label Learning with Missing Values using Combined Facial Action Unit Datasets
• Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion
• RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations
• Bayesian Convolutional Neural Networks for Seven Basic Facial Expression Classifications
• Machine vision detection to daily facial fatigue with a nonlocal 3D attention network
• Seven Basic Expression Recognition Using ResNet-18
• Causal affect prediction model using a facial image sequence
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild
• Deep Convolutional Neural Network Based Facial Expression Recognition in the Wild
• Interpretable Deep Neural Networks for Facial Expression and Dimensional Emotion Recognition in-the-wild
• A Multi-resolution Approach to Expression Recognition in the Wild
• Expression Recognition Analysis in the Wild
• Expression Recognition in the Wild Using Sequence Modeling
• Multi-Modal Continuous Valence And Arousal Prediction in the Wild Using Deep 3D Features and Sequence Modeling