Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition

Abstract

Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation and high-performing deep architectures. The deep feature representations of the audio and visual modalities are fused based on a model-level fusion strategy. A recurrent neural network is then used to capture the temporal dynamics. Our proposed approach substantially outperforms state-of-the-art approaches in predicting valence on the RECOLA dataset. Moreover, our proposed visual facial expression feature extraction network outperforms state-of-the-art results on the AffectNet and Google Facial Expression Comparison datasets.

8 pages, 3 figures, Pattern Recognition Letters

References in corpus (3)

• Distilling the Knowledge in a Neural Network
• Deep Learning for Human Affect Recognition: Insights and New Developments
• AudVowelConsNet: A Phoneme-Level Based Deep CNN Architecture for Clinical Depression Diagnosis

Cited by in corpus (9)

• Facial expression and attributes recognition based on multi-task learning of lightweight neural networks
• A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations
• Video-based Cross-modal Auxiliary Network for Multimodal Sentiment Analysis
• Versatile audio-visual learning for emotion recognition
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Unveiling the frontiers of deep learning: innovations shaping diverse domains
• Gaze-enhanced Crossmodal Embeddings for Emotion Recognition
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• Distilled Non-Semantic Speech Embeddings with Binary Neural Networks for Low-Resource Devices