Supervised Contrastive Learning for Affect Modelling

Abstract

Affect modeling is viewed, traditionally, as the process of mapping measurable affect manifestations from multiple modalities of user input to affect labels. That mapping is usually inferred through end-to-end (manifestation-to-affect) machine learning processes. What if, instead, one trains general, subject-invariant representations that consider affect information and then uses such representations to model affect? In this paper we assume that affect labels form an integral part, and not just the training signal, of an affect representation and we explore how the recent paradigm of contrastive learning can be employed to discover general high-level affect-infused representations for the purpose of modeling affect. We introduce three different supervised contrastive learning approaches for training representations that consider affect information. In this initial study we test the proposed methods for arousal prediction in the RECOLA dataset based on user information from multiple modalities. Results demonstrate the representation capacity of contrastive learning and its efficiency in boosting the accuracy of affect models. Beyond their evidenced higher performance compared to end-to-end arousal classification, the resulting representations are general-purpose and subject-agnostic, as training is guided though general affect information available in any multimodal corpus.

arXiv commentThis paper was accepted to ICMI 2022

References in corpus (7)

• Contrastive Representation Learning: A Framework and Review
• End-to-End Multimodal Emotion Recognition using Deep Neural Networks
• Contrastive Learning of Subject-Invariant EEG Representations for Cross-Subject Emotion Recognition
• A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeat
• Dynamic Difficulty Awareness Training for Continuous Emotion Prediction
• The Arousal video Game AnnotatIoN (AGAIN) Dataset
• Self-supervised Contrastive Learning of Multi-view Facial Expressions

Cited by in corpus (2)

• Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad Actions
• Privileged Contrastive Pretraining for Multimodal Affect Modelling