Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond

Abstract

Automatic understanding of human affect using visual signals is of great importance in everyday human-machine interactions. Appraising human emotional states, behaviors and reactions displayed in real-world settings, can be accomplished using latent continuous dimensions (e.g., the circumplex model of affect). Valence (i.e., how positive or negative is an emotion) & arousal (i.e., power of the activation of the emotion) constitute popular and effective affect representations. Nevertheless, the majority of collected datasets this far, although containing naturalistic emotional states, have been captured in highly controlled recording conditions. In this paper, we introduce the Aff-Wild benchmark for training and evaluating affect recognition algorithms. We also report on the results of the First Affect-in-the-wild Challenge that was organized in conjunction with CVPR 2017 on the Aff-Wild database and was the first ever challenge on the estimation of valence and arousal in-the-wild. Furthermore, we design and extensively train an end-to-end deep neural architecture which performs prediction of continuous emotion dimensions based on visual cues. The proposed deep learning architecture, AffWildNet, includes convolutional & recurrent neural network layers, exploiting the invariant properties of convolutional features, while also modeling temporal dynamics that arise in human behavior via the recurrent layers. The AffWildNet produced state-of-the-art results on the Aff-Wild Challenge. We then exploit the AffWild database for learning features, which can be used as priors for achieving best performances both for dimensional, as well as categorical emotion recognition, using the RECOLA, AFEW-VA and EmotiW datasets, compared to all other methods designed for the same goal. The database and emotion recognition models are available at http://ibug.doc.ic.ac.uk/resources/first-affect-wild-challenge.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• End-to-End Multimodal Emotion Recognition using Deep Neural Networks
• Survey of resampling techniques for improving classification performance in unbalanced datasets
• A Comprehensive Performance Evaluation of Deformable Face Tracking "In-the-Wild"
• Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video
• Facial Affect Estimation in the Wild Using Deep Residual and Convolutional Networks

Cited by in corpus (54)

• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset
• Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition
• Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
• A Multi-component CNN-RNN Approach for Dimensional Emotion Recognition in-the-wild
• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• Multitask Emotion Recognition with Incomplete Labels
• From the Lab to the Wild: Affect Modeling via Privileged Information
• GameVibe: A Multimodal Affective Game Corpus
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
• Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild
• FEAFA+: An Extended Well-Annotated Dataset for Facial Expression Analysis and 3D Facial Animation
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Deep Adaptation of Adult-Child Facial Expressions by Fusing Landmark Features
• Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark
• Feature Pyramid Network for Multi-task Affective Analysis
• Emotion Recognition for In-the-wild Videos
• Affect Expression Behaviour Analysis in the Wild using Spatio-Channel Attention and Complementary Context Information
• Action Units Recognition by Pairwise Deep Architecture
• $M^3$T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild
• MuSe 2020 -- The First International Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop
• Spatial and Temporal Networks for Facial Expression Recognition in the Wild Videos
• A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis
• Multi-label Relation Modeling in Facial Action Units Detection
• Inferring Dynamic Representations of Facial Actions from a Still Image
• Action Unit Detection with Joint Adaptive Attention and Graph Relation
• Multi-Label Class Balancing Algorithm for Action Unit Detection
• Lossless Attention in Convolutional Networks for Facial Expression Recognition in the Wild
• Emotion Generation and Recognition: A StarGAN Approach
• Aff-Wild Database and AffWildNet
• An Audio-Video Deep and Transfer Learning Framework for Multimodal Emotion Recognition in the wild
• Action Units Recognition Using Improved Pairwise Deep Architecture
• Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Multi-label Learning with Missing Values using Combined Facial Action Unit Datasets
• iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysis
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion
• Causal affect prediction model using a facial image sequence
• Seven Basic Expression Recognition Using ResNet-18
• Bayesian Convolutional Neural Networks for Seven Basic Facial Expression Classifications
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild
• RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations
• The FaceChannelS: Strike of the Sequences for the AffWild 2 Challenge
• Deep Convolutional Neural Network Based Facial Expression Recognition in the Wild
• Facial Emotion Recognition using Deep Residual Networks in Real-World Environments
• A Multi-resolution Approach to Expression Recognition in the Wild
• Expression Recognition Analysis in the Wild
• AI in Pursuit of Happiness, Finding Only Sadness: Multi-Modal Facial Emotion Recognition Challenge