Deep Learning for Human Affect Recognition: Insights and New Developments

Abstract

Automatic human affect recognition is a key step towards more natural human-computer interaction. Recent trends include recognition in the wild using a fusion of audiovisual and physiological sensors, a challenging setting for conventional machine learning algorithms. Since 2010, novel deep learning algorithms have been applied increasingly in this field. In this paper, we review the literature on human affect recognition between 2010 and 2017, with a special focus on approaches using deep neural networks. By classifying a total of 950 studies according to their usage of shallow or deep architectures, we are able to show a trend towards deep learning. Reviewing a subset of 233 studies that employ deep neural networks, we comprehensively quantify their applications in this field. We find that deep learning is used for learning of (i) spatial feature representations, (ii) temporal feature representations, and (iii) joint feature representations for multimodal sensor data. Exemplary state-of-the-art architectures illustrate the progress. Our findings show the role deep architectures will play in human affect recognition, and can serve as a reference point for researchers working on related applications.

To be published in IEEE Transactions on Affective Computing. 20 pages, 7 figures, 6 tables

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• End-to-End Multimodal Emotion Recognition using Deep Neural Networks
• Deep Learning using Linear Support Vector Machines
• Spatial-Temporal Recurrent Neural Network for Emotion Recognition
• Regularization for Deep Learning: A Taxonomy
• A Deep Learning Perspective on the Origin of Facial Expressions

Cited by in corpus (15)

• Deep Facial Expression Recognition: A Survey
• Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
• SVFAP: Self-supervised Video Facial Affect Perceiver
• OREBA: A Dataset for Objectively Recognizing Eating Behaviour and Associated Intake
• The Emotionally Intelligent Robot: Improving Social Navigation in Crowded Environments
• Gaze-enhanced Crossmodal Embeddings for Emotion Recognition
• Learning to Augment Expressions for Few-shot Fine-grained Facial Expression Recognition
• Deep Neural Baselines for Computational Paralinguistics
• Nonverbal Immediacy Analysis in Education: A Multimodal Computational Model
• MAFER: a Multi-resolution Approach to Facial Expression Recognition
• It's LeVAsa not LevioSA! Latent Encodings for Valence-Arousal Structure Alignment
• Towards Robust Deep Neural Networks for Affect and Depression Recognition from Speech
• Multi-Cue Adaptive Emotion Recognition Network
• Expression Recognition Analysis in the Wild
• A Multi-resolution Approach to Expression Recognition in the Wild