Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition

Abstract

Human emotion recognition plays an important role in human-computer interaction. In this paper, we present our approach to the Valence-Arousal (VA) Estimation Challenge, Expression (Expr) Classification Challenge, and Action Unit (AU) Detection Challenge of the 5th Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). Specifically, we propose a novel multi-modal fusion model that leverages Temporal Convolutional Networks (TCN) and Transformer to enhance the performance of continuous emotion recognition. Our model aims to effectively integrate visual and audio information for improved accuracy in recognizing emotions. Our model outperforms the baseline and ranks 3 in the Expression Classification challenge.

arXiv comment2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

References in corpus (9)

• EfficientNetV2: Smaller Models and Faster Training
• End-to-End Multimodal Emotion Recognition using Deep Neural Networks
• Parallel Spatio-Temporal Attention-Based TCN for Multivariate Time Series Prediction
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Two-Stream Aural-Visual Affect Analysis in the Wild
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition