From the Lab to the Wild: Affect Modeling via Privileged Information
arXiv:2305.10919 · doi:10.1109/TAFFC.2023.3265072
Abstract
How can we reliably transfer affect models trained in controlled laboratory conditions (in-vitro) to uncontrolled real-world settings (in-vivo)? The information gap between in-vitro and in-vivo applications defines a core challenge of affective computing. This gap is caused by limitations related to affect sensing including intrusiveness, hardware malfunctions and availability of sensors. As a response to these limitations, we introduce the concept of privileged information for operating affect models in real-world scenarios (in the wild). Privileged information enables affect models to be trained across multiple modalities available in a lab, and ignore, without significant performance drops, those modalities that are not available when they operate in the wild. Our approach is tested in two multimodal affect databases one of which is designed for testing models of affect in the wild. By training our affect models using all modalities and then using solely raw footage frames for testing the models, we reach the performance of models that fuse all available modalities for both training and testing. The results are robust across both classification and regression affect modeling tasks which are dominant paradigms in affective computing. Our findings make a decisive step towards realizing affect interaction in the wild.
arXiv comment13 pages, accepted for publication in IEEE Transactions on Affective Computing. arXiv admin note: text overlap with arXiv:2107.10552
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
- End-to-End Multimodal Emotion Recognition using Deep Neural Networks
- Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
- SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild
- Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
- A Deep Learning Perspective on the Origin of Facial Expressions
- Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
- The Pixels and Sounds of Emotion: General-Purpose Representations of Arousal in Games
- Dynamic Difficulty Awareness Training for Continuous Emotion Prediction
- Multimodal Emotion Recognition for One-Minute-Gradual Emotion Challenge