Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion

Abstract

We propose an audio-visual spatial-temporal deep neural network with: (1) a visual block containing a pretrained 2D-CNN followed by a temporal convolutional network (TCN); (2) an aural block containing several parallel TCNs; and (3) a leader-follower attentive fusion block combining the audio-visual information. The TCN with large history coverage enables our model to exploit spatial-temporal information within a much larger window length (i.e., 300) than that from the baseline and state-of-the-art methods (i.e., 36 or 48). The fusion block emphasizes the visual modality while exploits the noisy aural modality using the inter-modality attention mechanism. To make full use of the data and alleviate over-fitting, cross-validation is carried out on the training and validation set. The concordance correlation coefficient (CCC) centering is used to merge the results from each fold. On the test (validation) set of the Aff-Wild2 database, the achieved CCC is 0.463 (0.469) for valence and 0.492 (0.649) for arousal, which significantly outperforms the baseline method with the corresponding CCC of 0.200 (0.210) and 0.190 (0.230) for valence and arousal, respectively. The code is available at https://github.com/sucv/ABAW2.

arXiv comment8 pages, 2 figures, 2 tables, accepted to the ICCV 2021: 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW2)

References in corpus (10)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild
• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis
• Aff-Wild Database and AffWildNet