Gaze-enhanced Crossmodal Embeddings for Emotion Recognition
arXiv:2205.00129 · doi:10.1145/3530879
Abstract
Emotional expressions are inherently multimodal -- integrating facial behavior, speech, and gaze -- but their automatic recognition is often limited to a single modality, e.g. speech during a phone call. While previous work proposed crossmodal emotion embeddings to improve monomodal recognition performance, despite its importance, an explicit representation of gaze was not included. We propose a new approach to emotion recognition that incorporates an explicit representation of gaze in a crossmodal emotion embedding framework. We show that our method outperforms the previous state of the art for both audio-only and video-only emotion classification on the popular One-Minute Gradual Emotion Recognition dataset. Furthermore, we report extensive ablation experiments and provide detailed insights into the performance of different state-of-the-art gaze representations and integration strategies. Our results not only underline the importance of gaze for emotion recognition but also demonstrate a practical and highly effective approach to leveraging gaze information for this task.
References in corpus (7)
- Adam: A Method for Stochastic Optimization
- Appearance-Based Gaze Estimation in the Wild
- Deep Learning for Human Affect Recognition: Insights and New Developments
- Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
- Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction
- Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour
- Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect prediction