Deep Facial Expression Recognition: A Survey

Abstract

With the transition of facial expression recognition (FER) from laboratory-controlled to challenging in-the-wild conditions and the recent success of deep learning techniques in various fields, deep neural networks have increasingly been leveraged to learn discriminative representations for automatic FER. Recent deep FER systems generally focus on two important issues: overfitting caused by a lack of sufficient training data and expression-unrelated variations, such as illumination, head pose and identity bias. In this paper, we provide a comprehensive survey on deep FER, including datasets and algorithms that provide insights into these intrinsic problems. First, we describe the standard pipeline of a deep FER system with the related background knowledge and suggestions of applicable implementations for each stage. We then introduce the available datasets that are widely used in the literature and provide accepted data selection and evaluation principles for these datasets. For the state of the art in deep FER, we review existing novel deep neural networks and related training strategies that are designed for FER based on both static images and dynamic image sequences, and discuss their advantages and limitations. Competitive performances on widely used benchmarks are also summarized in this section. We then extend our survey to additional related issues and application scenarios. Finally, we review the remaining challenges and corresponding opportunities in this field as well as future directions for the design of robust deep FER systems.

References in corpus (42)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• Learning Face Representation from Scratch
• Going Deeper in Facial Expression Recognition using Deep Neural Networks
• Deep Learning using Linear Support Vector Machines
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Autoencoding beyond pixels using a learned similarity metric
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
• Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks
• Deep Learning for Human Affect Recognition: Insights and New Developments
• Facial Expression Recognition using Convolutional Neural Networks: State of the Art
• Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis
• AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
• Challenges in Representation Learning: A report on three machine learning contests
• A Deep Learning Perspective on the Origin of Facial Expressions
• Deep generative-contrastive networks for facial expression recognition
• Spatio-Temporal Facial Expression Recognition Using Convolutional Neural Networks and Conditional Random Fields
• Do We Really Need to Collect Millions of Faces for Effective Face Recognition?
• Semantic Facial Expression Editing using Autoencoded Flow
• Geometry-Contrastive GAN for Facial Expression Transfer
• EmotioNet Challenge: Recognition of facial expressions of emotion in the wild
• Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video
• Multi-task, multi-label and multi-domain learning with residual convolutional networks for emotion recognition
• Real-time emotion recognition for gaming using deep convolutional network features
• Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition?
• Learning Social Relation Traits from Face Images
• Deep Covariance Descriptors for Facial Expression Recognition
• Geometry Guided Adversarial Facial Expression Synthesis
• Photorealistic Facial Expression Synthesis by the Conditional Difference Adversarial Autoencoder
• VGAN-Based Image Representation Learning for Privacy-Preserving Facial Expression Recognition
• Differential Generative Adversarial Networks: Synthesizing Non-linear Facial Variations with Limited Number of Training Data
• Eyemotion: Classifying facial expressions in VR using eye-tracking cameras
• Accurate Facial Parts Localization and Deep Learning for 3D Facial Expression Recognition

Cited by in corpus (116)

• Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion
• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
• Medical Deep Learning -- A systematic Meta-Review
• Levels of explainable artificial intelligence for human-aligned conversational explanations
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• Fusing Body Posture with Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction
• Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition
• Dynamic Facial Expression Recognition under Partial Occlusion with Optical Flow Reconstruction
• Domain-Incremental Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition
• Disentangling Identity and Pose for Facial Expression Recognition
• Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition
• Facial-Sketch Synthesis: A New Challenge
• Sampling Strategies for GAN Synthetic Data
• Identity-Free Facial Expression Recognition using conditional Generative Adversarial Network
• Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher
• Affect-Driven Modelling of Robot Personality for Collaborative Human-Robot Interactions
• Uncovering the Limits of Text-based Emotion Detection
• Digital trace data collection through data donation
• SVFAP: Self-supervised Video Facial Affect Perceiver
• Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
• Automated Detection of Cat Facial Landmarks
• Multi-Branch Deep Radial Basis Function Networks for Facial Emotion Recognition
• Facial Expression Recognition with Deep Learning
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• Anchorage: Visual Analysis of Satisfaction in Customer Service Videos via Anchor Events
• Pose-adaptive Hierarchical Attention Network for Facial Expression Recognition
• The EMPATHIC Framework for Task Learning from Implicit Human Feedback
• Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey
• Unveiling the Human-like Similarities of Automatic Facial Expression Recognition: An Empirical Exploration through Explainable AI
• Empathetic Conversational Agents: Utilizing Neural and Physiological Signals for Enhanced Empathetic Interactions
• Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysis
• Towards Interpretability in Audio and Visual Affective Machine Learning: A Review
• Artificial Intelligence can Recognize Whether a Job Applicant is Selling and/or Lying According to Facial Expressions and Head Movements Much More Correctly Than Human Interviewers
• Do Deep Neural Networks Forget Facial Action Units? -- Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition
• Learning to Augment Expressions for Few-shot Fine-grained Facial Expression Recognition
• HighlightMe: Detecting Highlights from Human-Centric Videos
• Cascade EF-GAN: Progressive Facial Expression Editing with Local Focuses
• WiFE: WiFi and Vision based Intelligent Facial-Gesture Emotion Recognition
• EyeNet: A Multi-Task Network for Off-Axis Eye Gaze Estimation and User Understanding
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Participant Perceptions of a Robotic Coach Conducting Positive Psychology Exercises: A Qualitative Analysis
• Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• A Detailed Look At CNN-based Approaches In Facial Landmark Detection
• EmoTwiCS: A Corpus for Modelling Emotion Trajectories in Dutch Customer Service Dialogues on Twitter
• Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Papers
• A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles
• A Two-stage Multi-modal Affect Analysis Framework for Children with Autism Spectrum Disorder
• Facial Expression Recognition Using Disentangled Adversarial Learning
• A Facial Affect Analysis System for Autism Spectrum Disorder
• Micro-Expression Spotting: A Benchmark
• Learning Disentangled Expression Representations from Facial Images
• Towards Fair Affective Robotics: Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition
• DEEMO: De-identity Multimodal Emotion Recognition and Reasoning
• Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data
• Affective Image Content Analysis: Two Decades Review and New Perspectives
• Deep Multi-Facial Patches Aggregation Network For Facial Expression Recognition
• PropagationNet: Propagate Points to Curve to Learn Structure Information
• Adversarial Learning with Margin-based Triplet Embedding Regularization
• Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective
• Adaptively Enhancing Facial Expression Crucial Regions via Local Non-Local Joint Network
• Non-binary deep transfer learning for image classification
• Facial Expressions as a Vulnerability in Face Recognition
• DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization
• Video-based Facial Expression Recognition using Graph Convolutional Networks
• Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition
• MFEViT: A Robust Lightweight Transformer-based Network for Multimodal 2D+3D Facial Expression Recognition
• Facial Expression Recognition Using Human to Animated-Character Expression Translation
• AU-Guided Unsupervised Domain Adaptive Facial Expression Recognition
• Adversarial Graph Representation Adaptation for Cross-Domain Facial Expression Recognition
• Deep and Shallow Covariance Feature Quantization for 3D Facial Expression Recognition
• Pose-aware Adversarial Domain Adaptation for Personalized Facial Expression Recognition
• On the Pitfalls of Learning with Limited Data: A Facial Expression Recognition Case Study
• Exploring Facial Expressions and Affective Domains for Parkinson Detection
• 3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework using Fully 3D Convolutional Neural Networks
• Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features
• Multi-scale Transformer-based Network for Emotion Recognition from Multi Physiological Signals
• Bridging the gap between emotion and joint action
• Morphset:Augmenting categorical emotion datasets with dimensional affect labels using face morphing
• RealSmileNet: A Deep End-To-End Network for Spontaneous and Posed Smile Recognition
• Distill-2MD-MTL: Data Distillation based on Multi-Dataset Multi-Domain Multi-Task Frame Work to Solve Face Related Tasksks, Multi Task Learning, Semi-Supervised Learning
• Improving Facial Emotion Recognition through Dataset Merging and Balanced Training Strategies
• Computational Emotion Analysis From Images: Recent Advances and Future Directions
• Predicting Poverty Level from Satellite Imagery using Deep Neural Networks
• Selective Pseudo-Labeling with Reinforcement Learning for Semi-Supervised Domain Adaptation
• From Pixels to Words: Leveraging Explainability in Face Recognition through Interactive Natural Language Processing
• Identity-aware Facial Expression Recognition in Compressed Video
• Continual Learning for Affective Computing
• Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognition
• Coupling deep and handcrafted features to assess smile genuineness
• ComFace: Facial Representation Learning with Synthetic Data for Comparing Faces
• Assessing the Efficacy of Deep Learning Approaches for Facial Expression Recognition in Individuals with Intellectual Disabilities
• Towards Understanding Confusion and Affective States Under Communication Failures in Voice-Based Human-Machine Interaction
• Facial Expressions Recognition with Convolutional Neural Networks
• Uncover Common Facial Expressions in Terracotta Warriors: A Deep Learning Approach
• Deep Learning -- A first Meta-Survey of selected Reviews across Scientific Disciplines, their Commonalities, Challenges and Research Impact
• Variants of BERT, Random Forests and SVM approach for Multimodal Emotion-Target Sub-challenge
• Mutual Information Regularized Identity-aware Facial ExpressionRecognition in Compressed Video
• Point Adversarial Self Mining: A Simple Method for Facial Expression Recognition
• Oculum afficit: Ocular Affect Recognition
• Omni-supervised Facial Expression Recognition via Distilled Data
• RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations
• Deep Multi-Facial patches Aggregation Network for Expression Classification from Face Images
• Impact of facial landmark localization on facial expression recognition
• Landmark-Aware and Part-based Ensemble Transfer Learning Network for Facial Expression Recognition from Static images
• Towards Reading Beyond Faces for Sparsity-Aware 4D Affect Recognition
• Towards Unbiased Visual Emotion Recognition via Causal Intervention
• Deep learning for identification and face, gender, expression recognition under constraints
• Emotion-Aware Classroom Quality Assessment Leveraging IoT-Based Real-Time Student Monitoring
• Facial Expression Recognition using Deep Learning
• AI in Pursuit of Happiness, Finding Only Sadness: Multi-Modal Facial Emotion Recognition Challenge
• Domain Adaptation for Facial Expression Classifier via Domain Discrimination and Gradient Reversal
• Event-Level Emotion Recognition in the Wild Using Deep Facial Expression Analysis
• Toward Multimodal Modeling of Emotional Expressiveness
• Joint Deep Learning of Facial Expression Synthesis and Recognition