Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition

Abstract

Automatic understanding of human affect using visual signals is a problem that has attracted significant interest over the past 20 years. However, human emotional states are quite complex. To appraise such states displayed in real-world settings, we need expressive emotional descriptors that are capable of capturing and describing this complexity. The circumplex model of affect, which is described in terms of valence (i.e., how positive or negative is an emotion) and arousal (i.e., power of the activation of the emotion), can be used for this purpose. Recent progress in the emotion recognition domain has been achieved through the development of deep neural architectures and the availability of very large training databases. To this end, Aff-Wild has been the first large-scale "in-the-wild" database, containing around 1,200,000 frames. In this paper, we build upon this database, extending it with 260 more subjects and 1,413,000 new video frames. We call the union of Aff-Wild with the additional data, Aff-Wild2. The videos are downloaded from Youtube and have large variations in pose, age, illumination conditions, ethnicity and profession. Both database-specific as well as cross-database experiments are performed in this paper, by utilizing the Aff-Wild2, along with the RECOLA database. The developed deep neural architectures are based on the joint training of state-of-the-art convolutional and recurrent neural networks with attention mechanism; thus exploiting both the invariant properties of convolutional features, while modeling temporal dynamics that arise in human behaviour via the recurrent layers. The obtained results show premise for utilization of the extended Aff-Wild, as well as of the developed deep neural architectures for visual analysis of human behaviour in terms of continuous emotion dimensions.

References in corpus (3)

• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• A Multi-Task Learning & Generation Framework: Valence-Arousal, Action Units & Primary Expressions
• A Multi-component CNN-RNN Approach for Dimensional Emotion Recognition in-the-wild

Cited by in corpus (40)

• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Noisy Student Training using Body Language Dataset Improves Facial Expression Recognition
• A Multi-component CNN-RNN Approach for Dimensional Emotion Recognition in-the-wild
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• Pose-adaptive Hierarchical Attention Network for Facial Expression Recognition
• Multitask Emotion Recognition with Incomplete Labels
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Affect Expression Behaviour Analysis in the Wild using Spatio-Channel Attention and Complementary Context Information
• Emotion Recognition for In-the-wild Videos
• Facial Affect Recognition in the Wild Using Multi-Task Learning Convolutional Network
• Action Units Recognition by Pairwise Deep Architecture
• CorrLoss: Integrating Co-Occurrence Domain Knowledge for Affect Recognition
• $M^3$T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild
• Multi-modal Affect Analysis using standardized data within subjects in the Wild
• A Multi-term and Multi-task Analyzing Framework for Affective Analysis in-the-wild
• Multi-label Relation Modeling in Facial Action Units Detection
• Adversarial-based neural networks for affect estimations in the wild
• An Audio-Video Deep and Transfer Learning Framework for Multimodal Emotion Recognition in the wild
• Multi-Label Class Balancing Algorithm for Action Unit Detection
• Lossless Attention in Convolutional Networks for Facial Expression Recognition in the Wild
• Emotion Generation and Recognition: A StarGAN Approach
• Aff-Wild Database and AffWildNet
• Action Units Recognition Using Improved Pairwise Deep Architecture
• MAFER: a Multi-resolution Approach to Facial Expression Recognition
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Multi-label Learning with Missing Values using Combined Facial Action Unit Datasets
• Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Causal affect prediction model using a facial image sequence
• Deep Convolutional Neural Network Based Facial Expression Recognition in the Wild
• Expression Recognition Analysis in the Wild
• Interpretable Deep Neural Networks for Facial Expression and Dimensional Emotion Recognition in-the-wild
• Expression Recognition in the Wild Using Sequence Modeling
• Multi-Modal Continuous Valence And Arousal Prediction in the Wild Using Deep 3D Features and Sequence Modeling
• A Multi-resolution Approach to Expression Recognition in the Wild
• AffWild Net and Aff-Wild Database