Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study

Abstract

Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm, such as a DNN. MTL is based on the assumption that the tasks under consideration are related; therefore it exploits shared knowledge for improving performance on each individual task. Tasks are generally considered to be homogeneous, i.e., to refer to the same type of problem. Moreover, MTL is usually based on ground truth annotations with full, or partial overlap across tasks. In this work, we deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We explore task-relatedness as a means for co-training, in a weakly-supervised way, tasks that contain little, or even non-overlapping annotations. Task-relatedness is introduced in MTL, either explicitly through prior expert knowledge, or through data-driven studies. We propose a novel distribution matching approach, in which knowledge exchange is enabled between tasks, via matching of their predictions' distributions. Based on this approach, we build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks. We develop case studies for: i) continuous affect estimation, action unit detection, basic emotion recognition; ii) attribute detection, face identification. We illustrate that co-training via task relatedness alleviates negative transfer. Since FaceBehaviorNet learns features that encapsulate all aspects of facial behavior, we conduct zero-/few-shot learning to perform tasks beyond the ones that it has been trained for, such as compound emotion recognition. By conducting a very large experimental study, utilizing 10 databases, we illustrate that our approach outperforms, by large margins, the state-of-the-art in all tasks and in all databases, even in these which have not been used in its training.

arXiv commentarXiv admin note: text overlap with arXiv:2103.15792, arXiv:1910.11111

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild
• Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace
• Pareto Multi-Task Learning
• Registration-free Face-SSD: Single shot analysis of smiles, facial attributes, and affect in the wild
• EmotioNet Challenge: Recognition of facial expressions of emotion in the wild
• Understanding and Improving Information Transfer in Multi-Task Learning
• Attributes for Improved Attributes: A Multi-Task Network for Attribute Classification
• Two-level Attention with Two-stage Multi-task Learning for Facial Emotion Recognition

Cited by in corpus (22)

• Leveraging TCN and Transformer for effective visual-audio fusion in continuous emotion recognition
• Prior Aided Streaming Network for Multi-task Affective Recognitionat the 2nd ABAW2 Competition
• A Multi-modal and Multi-task Learning Method for Action Unit and Expression Recognition
• NAC-TCN: Temporal Convolutional Networks with Causal Dilated Neighborhood Attention for Emotion Understanding
• Emotion Recognition with Incomplete Labels Using Modified Multi-task Learning Technique
• Affect Expression Behaviour Analysis in the Wild using Consensual Collaborative Training
• Consensual Collaborative Training And Knowledge Distillation Based Facial Expression Recognition Under Noisy Annotations
• Feature Pyramid Network for Multi-task Affective Analysis
• Deep Adaptation of Adult-Child Facial Expressions by Fusing Landmark Features
• Personalised Federated Learning On Heterogeneous Feature Spaces
• Spatial and Temporal Networks for Facial Expression Recognition in the Wild Videos
• Multi-modal Affect Analysis using standardized data within subjects in the Wild
• A Multi-task Mean Teacher for Semi-supervised Facial Affective Behavior Analysis
• Action Unit Detection with Joint Adaptive Attention and Graph Relation
• Action Units Recognition Using Improved Pairwise Deep Architecture
• Continuous Emotion Recognition with Audio-visual Leader-follower Attentive Fusion
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Technical Report for Valence-Arousal Estimation in ABAW2 Challenge
• Causal affect prediction model using a facial image sequence
• Seven Basic Expression Recognition Using ResNet-18
• Bayesian Convolutional Neural Networks for Seven Basic Facial Expression Classifications
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild