J$\hat{\text{A}}$A-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention

Abstract

Facial action unit (AU) detection and face alignment are two highly correlated tasks, since facial landmarks can provide precise AU locations to facilitate the extraction of meaningful local features for AU detection. However, most existing AU detection works handle the two tasks independently by treating face alignment as a preprocessing, and often use landmarks to predefine a fixed region or attention for each AU. In this paper, we propose a novel end-to-end deep learning framework for joint AU detection and face alignment, which has not been explored before. In particular, multi-scale shared feature is learned firstly, and high-level feature of face alignment is fed into AU detection. Moreover, to extract precise local features, we propose an adaptive attention learning module to refine the attention map of each AU adaptively. Finally, the assembled local features are integrated with face alignment feature and global feature for AU detection. Extensive experiments demonstrate that our framework (i) significantly outperforms the state-of-the-art AU detection methods on the challenging BP4D, DISFA, GFT and BP4D+ benchmarks, (ii) can adaptively capture the irregular region of each AU, (iii) achieves competitive performance for face alignment, and (iv) also works well under partial occlusions and non-frontal poses. The code for our method is available at https://github.com/ZhiwenShao/PyTorch-JAANet.

arXiv commentThis paper is the extended version of arXiv:1803.05588, and is accepted by International Journal of Computer Vision

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• Learning Deep Representation for Face Alignment with Auxiliary Attributes
• Deep Adaptive Attention for Joint Facial Action Unit Detection and Face Alignment
• Facial Action Unit Detection Using Attention and Relation Learning
• Deep Multi-Center Learning for Face Alignment
• Simultaneous Facial Landmark Detection, Pose and Deformation Estimation under Facial Occlusion
• Learning deep representation from coarse to fine for face alignment

Cited by in corpus (19)

• Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition
• Facial Action Unit Detection via Adaptive Attention and Relation
• GeoConv: Geodesic Guided Convolution for Facial Action Unit Recognition
• Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
• MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
• Your "Attention" Deserves Attention: A Self-Diversified Multi-Channel Attention for Facial Action Analysis
• Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning
• MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection
• Let's Get the FACS Straight -- Reconstructing Obstructed Facial Features
• CorrLoss: Integrating Co-Occurrence Domain Knowledge for Affect Recognition
• Micro-Expression Recognition via Fine-Grained Dynamic Perception
• Using Positive Matching Contrastive Loss with Facial Action Units to mitigate bias in Facial Expression Recognition
• Beyond Overfitting: Doubly Adaptive Dropout for Generalizable AU Detection
• Face Super-Resolution with Progressive Embedding of Multi-scale Face Priors
• Symmetric Perception and Ordinal Regression for Detecting Scoliosis Natural Image
• Spatio-Temporal Analysis of Facial Actions using Lifecycle-Aware Capsule Networks
• Computational efficient deep neural network with difference attention maps for facial action unit detection
• HiCOMEX: Facial Action Unit Recognition Based on Hierarchy Intensity Distribution and COMEX Relation Learning
• BPLF: A Bi-Parallel Linear Flow Model for Facial Expression Generation from Emotion Set Images