Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network

Abstract

Automatic facial behavior analysis has a long history of studies in the intersection of computer vision, physiology and psychology. However it is only recently, with the collection of large-scale datasets and powerful machine learning methods such as deep neural networks, that automatic facial behavior analysis started to thrive. Three of its iconic tasks are automatic recognition of basic expressions (e.g. happy, sad, surprised), estimation of continuous emotions (e.g., valence and arousal), and detection of facial action units (activations of e.g. upper/inner eyebrows, nose wrinkles). Up until now these tasks have been mostly studied independently collecting a dataset for the task. We present the first and the largest study of all facial behaviour tasks learned jointly in a single multi-task, multi-domain and multi-label network, which we call FaceBehaviorNet. For this we utilize all publicly available datasets in the community (around 5M images) that study facial behaviour tasks in-the-wild. We demonstrate that training jointly an end-to-end network for all tasks has consistently better performance than training each of the single-task networks. Furthermore, we propose two simple strategies for coupling the tasks during training, co-annotation and distribution matching, and show the advantages of this approach. Finally we show that FaceBehaviorNet has learned features that encapsulate all aspects of facial behaviour, and can be successfully applied to perform tasks (compound emotion recognition) beyond the ones that it has been trained in a zero- and few-shot learning setting.

arXiv commentfiled as a patent

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• A Survey on Multi-Task Learning
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• EmotioNet Challenge: Recognition of facial expressions of emotion in the wild
• A Multi-Task Learning & Generation Framework: Valence-Arousal, Action Units & Primary Expressions
• Two-level Attention with Two-stage Multi-task Learning for Facial Emotion Recognition

Cited by in corpus (27)

• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition
• Multi-Branch Deep Radial Basis Function Networks for Facial Emotion Recognition
• A Multi-component CNN-RNN Approach for Dimensional Emotion Recognition in-the-wild
• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• Multitask Emotion Recognition with Incomplete Labels
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
• Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Feature Pyramid Network for Multi-task Affective Analysis
• Spatial and Temporal Networks for Facial Expression Recognition in the Wild Videos
• Multi-modal Affect Analysis using standardized data within subjects in the Wild
• Technical Report for Valence-Arousal Estimation on Affwild2 Dataset
• A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis
• A Multi-term and Multi-task Analyzing Framework for Affective Analysis in-the-wild
• Action Units Recognition Using Improved Pairwise Deep Architecture
• Action Unit Detection with Joint Adaptive Attention and Graph Relation
• Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion
• Technical Report for Valence-Arousal Estimation in ABAW2 Challenge
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations
• Bayesian Convolutional Neural Networks for Seven Basic Facial Expression Classifications
• Seven Basic Expression Recognition Using ResNet-18
• Causal affect prediction model using a facial image sequence
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild