Multi-Branch Deep Radial Basis Function Networks for Facial Emotion Recognition

Abstract

Emotion recognition (ER) from facial images is one of the landmark tasks in affective computing with major developments in the last decade. Initial efforts on ER relied on handcrafted features that were used to characterize facial images and then feed to standard predictive models. Recent methodologies comprise end-to-end trainable deep learning methods that simultaneously learn both, features and predictive model. Perhaps the most successful models are based on convolutional neural networks (CNNs). While these models have excelled at this task, they still fail at capturing local patterns that could emerge in the learning process. We hypothesize these patterns could be captured by variants based on locally weighted learning. Specifically, in this paper we propose a CNN based architecture enhanced with multiple branches formed by radial basis function (RBF) units that aims at exploiting local information at the final stage of the learning process. Intuitively, these RBF units capture local patterns shared by similar instances using an intermediate representation, then the outputs of the RBFs are feed to a softmax layer that exploits this information to improve the predictive performance of the model. This feature could be particularly advantageous in ER as cultural / ethnicity differences may be identified by the local units. We evaluate the proposed method in several ER datasets and show the proposed methodology achieves state-of-the-art in some of them, even when we adopt a pre-trained VGG-Face model as backbone. We show it is the incorporation of local information what makes the proposed model competitive.

arXiv commentNeural Comput & Applic (2021)

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Deep Facial Expression Recognition: A Survey
• Going Deeper in Facial Expression Recognition using Deep Neural Networks
• Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
• Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
• Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines
• A review of learning vector quantization classifiers
• Landmark Guidance Independent Spatio-channel Attention and Complementary Context Information based Facial Expression Recognition
• Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network
• A Real Time Facial Expression Classification System Using Local Binary Patterns
• Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network
• Recognition of facial expressions based on salient geometric features and support vector machines
• Deep-RBF Networks Revisited: Robust Classification with Rejection
• Deep k-NN for Noisy Labels