AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild

Abstract

Automated affective computing in the wild setting is a challenging problem in computer vision. Existing annotated databases of facial expressions in the wild are small and mostly cover discrete emotions (aka the categorical model). There are very limited annotated facial databases for affective computing in the continuous dimensional model (e.g., valence and arousal). To meet this need, we collected, annotated, and prepared for public distribution a new database of facial emotions in the wild (called AffectNet). AffectNet contains more than 1,000,000 facial images from the Internet by querying three major search engines using 1250 emotion related keywords in six different languages. About half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. AffectNet is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. Two baseline deep neural networks are used to classify images in the categorical model and predict the intensity of valence and arousal. Various evaluation metrics show that our deep neural network baselines can perform better than conventional machine learning methods and off-the-shelf facial expression recognition systems.

arXiv commentIEEE Transactions on Affective Computing, 2017

References in corpus (7)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Going Deeper in Facial Expression Recognition using Deep Neural Networks
• Deep Learning using Linear Support Vector Machines
• AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
• Facial Expression Recognition from World Wild Web
• Bidirectional Warping of Active Appearance Model

Cited by in corpus (172)

• Deep Facial Expression Recognition: A Survey
• Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
• Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion
• Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy
• Deep Learning for Human Affect Recognition: Insights and New Developments
• Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
• Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot Study
• Facial expression and attributes recognition based on multi-task learning of lightweight neural networks
• K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations
• Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition
• A Deeper Look at Facial Expression Dataset Bias
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic Review
• SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation
• Landmark Guidance Independent Spatio-channel Attention and Complementary Context Information based Facial Expression Recognition
• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Self-supervised learning of a facial attribute embedding from video
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset
• Stimuli-Aware Visual Emotion Analysis
• Emotional Speech-Driven Animation with Content-Emotion Disentanglement
• PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression
• Fusing Body Posture with Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction
• Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition
• From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos
• Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis
• Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition
• Learning to Amend Facial Expression Representation via De-albino and Affinity
• Deep Feature Space: A Geometrical Perspective
• HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild
• Disentangling Identity and Pose for Facial Expression Recognition
• PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship
• Suppressing Uncertainties for Large-Scale Facial Expression Recognition
• Registration-free Face-SSD: Single shot analysis of smiles, facial attributes, and affect in the wild
• Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition
• Sampling Strategies for GAN Synthetic Data
• The Arousal video Game AnnotatIoN (AGAIN) Dataset
• Semantic Neighborhood-Aware Deep Facial Expression Recognition
• Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher
• BReG-NeXt: Facial Affect Computing Using Adaptive Residual Networks With Bounded Gradient
• Deep Temporal Analysis for Non-Acted Body Affect Recognition
• GANonymization: A GAN-based Face Anonymization Framework for Preserving Emotional Expressions
• OREBA: A Dataset for Objectively Recognizing Eating Behaviour and Associated Intake
• Versatile audio-visual learning for emotion recognition
• Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate
• Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
• Facial Expression Recognition with Deep Learning
• DAiSEE: Towards User Engagement Recognition in the Wild
• Photo-Realistic Facial Details Synthesis from Single Image
• Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT
• Toward Fine-grained Facial Expression Manipulation
• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• Enhancing Micro Gesture Recognition for Emotion Understanding via Context-aware Visual-Text Contrastive Learning
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey
• Uncertain Label Correction via Auxiliary Action Unit Graphs for Facial Expression Recognition
• Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad Actions
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Learning Disentangled Representations with Reference-Based Variational Autoencoders
• Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers
• Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches
• E-ffective: A Visual Analytic System for Exploring the Emotion and Effectiveness of Inspirational Speeches
• Contrastive Learning of Person-independent Representations for Facial Action Unit Detection
• Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild
• Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey
• Facial recognition technology and human raters can predict political orientation from images of expressionless faces even when controlling for demographics and self-presentation
• US-GAN: On the importance of Ultimate Skip Connection for Facial Expression Synthesis
• Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study
• Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS
• Towards a General Deep Feature Extractor for Facial Expression Recognition
• Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition
• Understanding and Mitigating Annotation Bias in Facial Expression Recognition
• Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing
• GameVibe: A Multimodal Affective Game Corpus
• XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networks
• Do Deep Neural Networks Forget Facial Action Units? -- Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition
• AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels
• A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection
• Developing a Data-Driven Categorical Taxonomy of Emotional Expressions in Real World Human Robot Interactions
• Synthetic Thermal and RGB Videos for Automatic Pain Assessment utilizing a Vision-MLP Architecture
• Learning to Augment Expressions for Few-shot Fine-grained Facial Expression Recognition
• Cascade EF-GAN: Progressive Facial Expression Editing with Local Focuses
• Measuring Student Behavioral Engagement using Histogram of Actions
• Occlusion-Adaptive Deep Network for Robust Facial Expression Recognition
• Responsible AI: Gender bias assessment in emotion recognition
• Facial Motion Prior Networks for Facial Expression Recognition
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• ARBEx: Attentive Feature Extraction with Reliability Balancing for Robust Facial Expression Learning
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Less can be more: representational vs. stereotypical gender bias in facial expression recognition
• FE-Adapter: Adapting Image-based Emotion Classifiers to Videos
• FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply
• DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild
• GReFEL: Geometry-Aware Reliable Facial Expression Learning under Bias and Imbalanced Data Distribution
• Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark
• Did the Model Change? Efficiently Assessing Machine Learning API Shifts
• Feature Pyramid Network for Multi-task Affective Analysis
• Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition
• Automatic Extraction of Road Networks by using Teacher-Student Adaptive Structural Deep Belief Network and Its Application to Landslide Disaster
• A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles
• Recognizing Micro-Expression in Video Clip with Adaptive Key-Frame Mining
• Cross-modal Deep Face Normals with Deactivable Skip Connections
• Persuasive Faces: Generating Faces in Advertisements
• Emotion Recognition for In-the-wild Videos
• Revisiting Few-Shot Learning for Facial Expression Recognition
• A Facial Affect Analysis System for Autism Spectrum Disorder
• Affect Expression Behaviour Analysis in the Wild using Spatio-Channel Attention and Complementary Context Information
• Sentiment-based Engagement Strategies for intuitive Human-Robot Interaction
• Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data
• Experimental Investigation of Trust in Anthropomorphic Agents as Task Partners
• $M^3$T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild
• Headset: Human emotion awareness under partial occlusions multimodal dataset
• Learning Disentangled Expression Representations from Facial Images
• Towards a General Model of Knowledge for Facial Analysis by Multi-Source Transfer Learning
• Exploring the contextual factors affecting multimodal emotion recognition in videos
• Facial Expressions as a Vulnerability in Face Recognition
• Adaptively Enhancing Facial Expression Crucial Regions via Local Non-Local Joint Network
• SLYKLatent: A Learning Framework for Gaze Estimation Using Deep Facial Feature Learning
• AU-Guided Unsupervised Domain Adaptive Facial Expression Recognition
• Pose-aware Adversarial Domain Adaptation for Personalized Facial Expression Recognition
• Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition
• FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model
• MFEViT: A Robust Lightweight Transformer-based Network for Multimodal 2D+3D Facial Expression Recognition
• A Personalized Affective Memory Neural Model for Improving Emotion Recognition
• Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features
• Lossless Attention in Convolutional Networks for Facial Expression Recognition in the Wild
• Complex Facial Expression Recognition Using Deep Knowledge Distillation of Basic Features
• Imponderous Net for Facial Expression Recognition in the Wild
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation
• FreeAvatar: Robust 3D Facial Animation Transfer by Learning an Expression Foundation Model
• EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment
• VAD-Net: Multidimensional Facial Expression Recognition in Intelligent Education System
• Detecting Interlocutor Confusion in Situated Human-Avatar Dialogue: A Pilot Study
• CG-MER: A Card Game-based Multimodal dataset for Emotion Recognition
• DualTake: Predicting Takeovers across Mobilities for Future Personalized Mobility Services
• Morphset:Augmenting categorical emotion datasets with dimensional affect labels using face morphing
• It's LeVAsa not LevioSA! Latent Encodings for Valence-Arousal Structure Alignment
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Hyperparameters optimization for Deep Learning based emotion prediction for Human Robot Interaction
• Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural Networks
• Technical Report for Valence-Arousal Estimation in ABAW2 Challenge
• The Many Moods of Emotion
• An Efficient Integration of Disentangled Attended Expression and Identity FeaturesFor Facial Expression Transfer andSynthesis
• Causal affect prediction model using a facial image sequence
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild
• Semi-Supervised Adversarial Discriminative Domain Adaptation
• A Distillation Learning Model of Adaptive Structural Deep Belief Network for AffectNet: Facial Expression Image Database
• DSAP: Analyzing Bias Through Demographic Comparison of Datasets
• Simulation of Human and Artificial Emotion (SHArE)
• LEED: Label-Free Expression Editing via Disentanglement
• Which Factors Predict the Chat Experience of a Natural Language Generation Dialogue Service?
• AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition
• Challenges and opportunities in portraying emotion in generated sign language
• RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations
• Emotion Transfer Using Vector-Valued Infinite Task Learning
• Oculum afficit: Ocular Affect Recognition
• Uncover Common Facial Expressions in Terracotta Warriors: A Deep Learning Approach
• ROSbag-based Multimodal Affective Dataset for Emotional and Cognitive States
• Weaknesses of Facial Emotion Recognition Systems
• Disambiguating Affective Stimulus Associations for Robot Perception and Dialogue
• Domain Adaptation for Facial Expression Classifier via Domain Discrimination and Gradient Reversal
• 3D Dense Geometry-Guided Facial Expression Synthesis by Adversarial Learning
• Re-learning of Child Model for Misclassified data by using KL Divergence in AffectNet: A Database for Facial Expression
• AI in Pursuit of Happiness, Finding Only Sadness: Multi-Modal Facial Emotion Recognition Challenge
• End-to-End Facial Expression Detection in Long Videos
• Continuous learning of face attribute synthesis
• Human Comfortability Index Estimation in Industrial Human-Robot Collaboration Task
• Affect-DML: Context-Aware One-Shot Recognition of Human Affect using Deep Metric Learning
• Harnessing Geometric Constraints from Emotion Labels to improve Face Verification