A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition

Abstract

Analyzing human affect is vital for human-computer interaction systems. Most methods are developed in restricted scenarios which are not practical for in-the-wild settings. The Affective Behavior Analysis in-the-wild (ABAW) 2021 Contest provides a benchmark for this in-the-wild problem. In this paper, we introduce a multi-modal and multi-task learning method by using both visual and audio information. We use both AU and expression annotations to train the model and apply a sequence model to further extract associations between video frames. We achieve an AU score of 0.712 and an expression score of 0.477 on the validation set. These results demonstrate the effectiveness of our approach in improving model performance.

arXiv comment5 pages, 3 figures, 2 tables

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• An overview of gradient descent optimization algorithms
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Partial FC: Training 10 Million Identities on a Single Machine

Cited by in corpus (6)

• Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition
• Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild