Local Learning with Deep and Handcrafted Features for Facial Expression Recognition

Abstract

We present an approach that combines automatic features learned by convolutional neural networks (CNN) and handcrafted features computed by the bag-of-visual-words (BOVW) model in order to achieve state-of-the-art results in facial expression recognition. To obtain automatic features, we experiment with multiple CNN architectures, pre-trained models and training procedures, e.g. Dense-Sparse-Dense. After fusing the two types of features, we employ a local learning framework to predict the class label for each test image. The local learning framework is based on three steps. First, a k-nearest neighbors model is applied in order to select the nearest training samples for an input test image. Second, a one-versus-all Support Vector Machines (SVM) classifier is trained on the selected training samples. Finally, the SVM classifier is used to predict the class label only for the test image it was trained for. Although we have used local learning in combination with handcrafted features in our previous work, to the best of our knowledge, local learning has never been employed in combination with deep features. The experiments on the 2013 Facial Expression Recognition (FER) Challenge data set, the FER+ data set and the AffectNet data set demonstrate that our approach achieves state-of-the-art results. With a top accuracy of 75.42% on FER 2013, 87.76% on the FER+, 59.58% on AffectNet 8-way classification and 63.31% on AffectNet 7-way classification, we surpass the state-of-the-art methods by more than 1% on all data sets.

Accepted in IEEE Access

References in corpus (4)

• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• Facial Expression Recognition Using Enhanced Deep 3D Convolutional Neural Networks
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Improving Bag-of-Visual-Words Towards Effective Facial Expressive Image Classification

Cited by in corpus (21)

• Deep Facial Expression Recognition: A Survey
• Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
• Facial Emotion Recognition: State of the Art Performance on FER2013
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Learning to Amend Facial Expression Representation via De-albino and Affinity
• HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild
• Disentangling Identity and Pose for Facial Expression Recognition
• EmoNeXt: an Adapted ConvNeXt for Facial Emotion Recognition
• Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition
• Multi-Branch Deep Radial Basis Function Networks for Facial Emotion Recognition
• Towards a General Deep Feature Extractor for Facial Expression Recognition
• A Fine-Grained Facial Expression Database for End-to-End Multi-Pose Facial Expression Recognition
• Learning to Augment Expressions for Few-shot Fine-grained Facial Expression Recognition
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Lossless Attention in Convolutional Networks for Facial Expression Recognition in the Wild
• MAFER: a Multi-resolution Approach to Facial Expression Recognition
• Hey Human, If your Facial Emotions are Uncertain, You Should Use Bayesian Neural Networks!
• Convolutional Neural Networks for User Identificationbased on Motion Sensors Represented as Image
• State of the Art: Face Recognition
• Continuous Trade-off Optimization between Fast and Accurate Deep Face Detectors