AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild

Abstract

Automated affective computing in the wild setting is a challenging problem in computer vision. Existing annotated databases of facial expressions in the wild are small and mostly cover discrete emotions (aka the categorical model). There are very limited annotated facial databases for affective computing in the continuous dimensional model (e.g., valence and arousal). To meet this need, we collected, annotated, and prepared for public distribution a new database of facial emotions in the wild (called AffectNet). AffectNet contains more than 1,000,000 facial images from the Internet by querying three major search engines using 1250 emotion related keywords in six different languages. About half of the retrieved images were manually annotated for the presence of seven discrete facial expressions and the intensity of valence and arousal. AffectNet is by far the largest database of facial expression, valence, and arousal in the wild enabling research in automated facial expression recognition in two different emotion models. Two baseline deep neural networks are used to classify images in the categorical model and predict the intensity of valence and arousal. Various evaluation metrics show that our deep neural network baselines can perform better than conventional machine learning methods and off-the-shelf facial expression recognition systems.

IEEE Transactions on Affective Computing, 2017

References in corpus (1)

• Caffe: Convolutional Architecture for Fast Feature Embedding

Cited by in corpus (171)

• Deep Facial Expression Recognition: A Survey
• Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
• Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion
• Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy
• Deep Learning for Human Affect Recognition: Insights and New Developments
• Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
• Artificial Emotional Intelligence in Socially Assistive Robots for Older Adults: A Pilot Study
• Facial expression and attributes recognition based on multi-task learning of lightweight neural networks
• K-EmoCon, a multimodal sensor dataset for continuous emotion recognition in naturalistic conversations
• Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition
• A Deeper Look at Facial Expression Dataset Bias
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic Review
• SwinFace: A Multi-task Transformer for Face Recognition, Expression Recognition, Age Estimation and Attribute Estimation
• Landmark Guidance Independent Spatio-channel Attention and Complementary Context Information based Facial Expression Recognition
• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• Self-supervised learning of a facial attribute embedding from video
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study
• Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Stimuli-Aware Visual Emotion Analysis
• Modeling emotion in complex stories: the Stanford Emotional Narratives Dataset
• Fusing Body Posture with Facial Expressions for Joint Recognition of Affect in Child-Robot Interaction
• PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression
• Emotional Speech-Driven Animation with Content-Emotion Disentanglement
• Exploring Emotion Features and Fusion Strategies for Audio-Video Emotion Recognition
• From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videos
• Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis
• Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition
• Learning to Amend Facial Expression Representation via De-albino and Affinity
• Deep Feature Space: A Geometrical Perspective
• HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild
• Disentangling Identity and Pose for Facial Expression Recognition
• PersEmoN: A Deep Network for Joint Analysis of Apparent Personality, Emotion and Their Relationship
• Suppressing Uncertainties for Large-Scale Facial Expression Recognition
• Registration-free Face-SSD: Single shot analysis of smiles, facial attributes, and affect in the wild
• Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition
• Sampling Strategies for GAN Synthetic Data
• The Arousal video Game AnnotatIoN (AGAIN) Dataset
• Semantic Neighborhood-Aware Deep Facial Expression Recognition
• Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher
• BReG-NeXt: Facial Affect Computing Using Adaptive Residual Networks With Bounded Gradient
• Deep Temporal Analysis for Non-Acted Body Affect Recognition
• GANonymization: A GAN-based Face Anonymization Framework for Preserving Emotional Expressions
• Versatile audio-visual learning for emotion recognition
• OREBA: A Dataset for Objectively Recognizing Eating Behaviour and Associated Intake
• Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate
• Region Attention Networks for Pose and Occlusion Robust Facial Expression Recognition
• Facial Expression Recognition with Deep Learning
• Photo-Realistic Facial Details Synthesis from Single Image
• Toward Fine-grained Facial Expression Manipulation
• Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT
• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• Enhancing Micro Gesture Recognition for Emotion Understanding via Context-aware Visual-Text Contrastive Learning
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey
• Predicting Player Engagement in Tom Clancy's The Division 2: A Multimodal Approach via Pixels and Gamepad Actions
• Uncertain Label Correction via Auxiliary Action Unit Graphs for Facial Expression Recognition
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Emotion Separation and Recognition from a Facial Expression by Generating the Poker Face with Vision Transformers
• Learning Disentangled Representations with Reference-Based Variational Autoencoders
• Automated Deception Detection from Videos: Using End-to-End Learning Based High-Level Features and Classification Approaches
• Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey
• Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild
• Contrastive Learning of Person-independent Representations for Facial Action Unit Detection
• Facial recognition technology and human raters can predict political orientation from images of expressionless faces even when controlling for demographics and self-presentation
• E-ffective: A Visual Analytic System for Exploring the Emotion and Effectiveness of Inspirational Speeches
• Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS
• Improved Digital Therapy for Developmental Pediatrics Using Domain-Specific Artificial Intelligence: Machine Learning Study
• US-GAN: On the importance of Ultimate Skip Connection for Facial Expression Synthesis
• Towards a General Deep Feature Extractor for Facial Expression Recognition
• Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition
• Bounded Residual Gradient Networks (BReG-Net) for Facial Affect Computing
• Understanding and Mitigating Annotation Bias in Facial Expression Recognition
• Do Deep Neural Networks Forget Facial Action Units? -- Exploring the Effects of Transfer Learning in Health Related Facial Expression Recognition
• Learning to Augment Expressions for Few-shot Fine-grained Facial Expression Recognition
• Developing a Data-Driven Categorical Taxonomy of Emotional Expressions in Real World Human Robot Interactions
• AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels
• A Review of Deep Learning Approaches for Non-Invasive Cognitive Impairment Detection
• Synthetic Thermal and RGB Videos for Automatic Pain Assessment utilizing a Vision-MLP Architecture
• GameVibe: A Multimodal Affective Game Corpus
• XnODR and XnIDR: Two Accurate and Fast Fully Connected Layers For Convolutional Neural Networks
• Responsible AI: Gender bias assessment in emotion recognition
• Measuring Student Behavioral Engagement using Histogram of Actions
• Cascade EF-GAN: Progressive Facial Expression Editing with Local Focuses
• Occlusion-Adaptive Deep Network for Robust Facial Expression Recognition
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• GReFEL: Geometry-Aware Reliable Facial Expression Learning under Bias and Imbalanced Data Distribution
• FE-Adapter: Adapting Image-based Emotion Classifiers to Videos
• Facial Motion Prior Networks for Facial Expression Recognition
• DFEW: A Large-Scale Database for Recognizing Dynamic Facial Expressions in the Wild
• Less can be more: representational vs. stereotypical gender bias in facial expression recognition
• FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply
• ARBEx: Attentive Feature Extraction with Reliability Balancing for Robust Facial Expression Learning
• Pairwise Emotional Relationship Recognition in Drama Videos: Dataset and Benchmark
• Automatic Extraction of Road Networks by using Teacher-Student Adaptive Structural Deep Belief Network and Its Application to Landslide Disaster
• Cross-modal Deep Face Normals with Deactivable Skip Connections
• A Spontaneous Driver Emotion Facial Expression (DEFE) Dataset for Intelligent Vehicles
• Feature Pyramid Network for Multi-task Affective Analysis
• Did the Model Change? Efficiently Assessing Machine Learning API Shifts
• Recognizing Micro-Expression in Video Clip with Adaptive Key-Frame Mining
• Learning Vision Transformer with Squeeze and Excitation for Facial Expression Recognition
• Revisiting Few-Shot Learning for Facial Expression Recognition
• Persuasive Faces: Generating Faces in Advertisements
• Sentiment-based Engagement Strategies for intuitive Human-Robot Interaction
• A Facial Affect Analysis System for Autism Spectrum Disorder
• Emotion Recognition for In-the-wild Videos
• Affect Expression Behaviour Analysis in the Wild using Spatio-Channel Attention and Complementary Context Information
• Headset: Human emotion awareness under partial occlusions multimodal dataset
• Learning Disentangled Expression Representations from Facial Images
• Static for Dynamic: Towards a Deeper Understanding of Dynamic Facial Expressions Using Static Expression Data
• $M^3$T: Multi-Modal Continuous Valence-Arousal Estimation in the Wild
• Experimental Investigation of Trust in Anthropomorphic Agents as Task Partners
• Facial Expressions as a Vulnerability in Face Recognition
• SLYKLatent: A Learning Framework for Gaze Estimation Using Deep Facial Feature Learning
• Adaptively Enhancing Facial Expression Crucial Regions via Local Non-Local Joint Network
• Exploring the contextual factors affecting multimodal emotion recognition in videos
• Towards a General Model of Knowledge for Facial Analysis by Multi-Source Transfer Learning
• FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model
• Label Distribution Amendment with Emotional Semantic Correlations for Facial Expression Recognition
• MFEViT: A Robust Lightweight Transformer-based Network for Multimodal 2D+3D Facial Expression Recognition
• Pose-aware Adversarial Domain Adaptation for Personalized Facial Expression Recognition
• A Personalized Affective Memory Neural Model for Improving Emotion Recognition
• AU-Guided Unsupervised Domain Adaptive Facial Expression Recognition
• Efficient Facial Expression Analysis For Dimensional Affect Recognition Using Geometric Features
• Complex Facial Expression Recognition Using Deep Knowledge Distillation of Basic Features
• Imponderous Net for Facial Expression Recognition in the Wild
• Lossless Attention in Convolutional Networks for Facial Expression Recognition in the Wild
• Technical Report for Valence-Arousal Estimation in ABAW2 Challenge
• CG-MER: A Card Game-based Multimodal dataset for Emotion Recognition
• The Many Moods of Emotion
• Efficient Facial Feature Learning with Wide Ensemble-based Convolutional Neural Networks
• Hyperparameters optimization for Deep Learning based emotion prediction for Human Robot Interaction
• It's LeVAsa not LevioSA! Latent Encodings for Valence-Arousal Structure Alignment
• VAD-Net: Multidimensional Facial Expression Recognition in Intelligent Education System
• FreeAvatar: Robust 3D Facial Animation Transfer by Learning an Expression Foundation Model
• DualTake: Predicting Takeovers across Mobilities for Future Personalized Mobility Services
• Facial Emotion Recognition with Noisy Multi-task Annotations
• Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment
• Morphset:Augmenting categorical emotion datasets with dimensional affect labels using face morphing
• Detecting Interlocutor Confusion in Situated Human-Avatar Dialogue: A Pilot Study
• An Efficient Integration of Disentangled Attended Expression and Identity FeaturesFor Facial Expression Transfer andSynthesis
• Semi-Supervised Adversarial Discriminative Domain Adaptation
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild
• Causal affect prediction model using a facial image sequence
• Uncover Common Facial Expressions in Terracotta Warriors: A Deep Learning Approach
• A Distillation Learning Model of Adaptive Structural Deep Belief Network for AffectNet: Facial Expression Image Database
• Oculum afficit: Ocular Affect Recognition
• AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition
• Emotion Transfer Using Vector-Valued Infinite Task Learning
• Simulation of Human and Artificial Emotion (SHArE)
• Which Factors Predict the Chat Experience of a Natural Language Generation Dialogue Service?
• DSAP: Analyzing Bias Through Demographic Comparison of Datasets
• LEED: Label-Free Expression Editing via Disentanglement
• RAF-AU Database: In-the-Wild Facial Expressions with Subjective Emotion Judgement and Objective AU Annotations
• Challenges and opportunities in portraying emotion in generated sign language
• Re-learning of Child Model for Misclassified data by using KL Divergence in AffectNet: A Database for Facial Expression
• 3D Dense Geometry-Guided Facial Expression Synthesis by Adversarial Learning
• Human Comfortability Index Estimation in Industrial Human-Robot Collaboration Task
• Affect-DML: Context-Aware One-Shot Recognition of Human Affect using Deep Metric Learning
• ROSbag-based Multimodal Affective Dataset for Emotional and Cognitive States
• Continuous learning of face attribute synthesis
• Domain Adaptation for Facial Expression Classifier via Domain Discrimination and Gradient Reversal
• End-to-End Facial Expression Detection in Long Videos
• Disambiguating Affective Stimulus Associations for Robot Perception and Dialogue
• Weaknesses of Facial Emotion Recognition Systems
• Harnessing Geometric Constraints from Emotion Labels to improve Face Verification
• AI in Pursuit of Happiness, Finding Only Sadness: Multi-Modal Facial Emotion Recognition Challenge