PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship

Abstract

Apparent personality and emotion analysis are both central to affective computing. Existing works solve them individually. In this paper we investigate if such high-level affect traits and their relationship can be jointly learned from face images in the wild. To this end, we introduce PersEmoN, an end-to-end trainable and deep Siamese-like network. It consists of two convolutional network branches, one for emotion and the other for apparent personality. Both networks share their bottom feature extraction module and are optimized within a multi-task learning framework. Emotion and personality networks are dedicated to their own annotated dataset. Furthermore, an adversarial-like loss function is employed to promote representation coherence among heterogeneous dataset sources. Based on this, we also explore the emotion-to-apparent-personality relationship. Extensive experiments demonstrate the effectiveness of PersEmoN.

arXiv commentAccepted to IEEE Transactions on Affective Computing

References in corpus (11)

• Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• DeepID3: Face Recognition with Very Deep Neural Networks
• SphereFace: Deep Hypersphere Embedding for Face Recognition
• Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition
• Explaining First Impressions: Modeling, Recognizing, and Explaining Apparent Personality from Videos
• First Impressions: A Survey on Vision-Based Apparent Personality Trait Analysis
• A Deep Network for Arousal-Valence Emotion Prediction with Acoustic-Visual Cues
• Facial Affect Estimation in the Wild Using Deep Residual and Convolutional Networks
• AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups

Cited by in corpus (8)

• SVFAP: Self-supervised Video Facial Affect Perceiver
• Personality-Driven Social Multimedia Content Recommendation
• Semi-Supervised Self-Taught Deep Learning for Finger Bones Segmentation
• Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches
• A Survey on Personality-Aware Recommendation Systems
• Psycholinguistic Tripartite Graph Network for Personality Detection
• Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition
• Being the center of attention: A Person-Context CNN framework for Personality Recognition