AU R-CNN: Encoding Expert Prior Knowledge into R-CNN for Action Unit Detection

Abstract

Detecting action units (AUs) on human faces is challenging because various AUs make subtle facial appearance change over various regions at different scales. Current works have attempted to recognize AUs by emphasizing important regions. However, the incorporation of expert prior knowledge into region definition remains under-exploited, and current AU detection approaches do not use regional convolutional neural networks (R-CNN) with expert prior knowledge to directly focus on AU-related regions adaptively. By incorporating expert prior knowledge, we propose a novel R-CNN based model named AU R-CNN. The proposed solution offers two main contributions: (1) AU R-CNN directly observes different facial regions, where various AUs are located. Specifically, we define an AU partition rule which encodes the expert prior knowledge into the region definition and RoI-level label definition. This design produces considerably better detection performance than existing approaches. (2) We integrate various dynamic models (including convolutional long short-term memory, two stream network, conditional random field, and temporal action localization network) into AU R-CNN and then investigate and analyze the reason behind the performance of dynamic models. Experiment results demonstrate that \textit{only} static RGB image information and no optical flow-based AU R-CNN surpasses the one fused with dynamic models. AU R-CNN is also superior to traditional CNNs that use the same backbone on varying image resolutions. State-of-the-art recognition performance of AU detection is achieved. The complete network is end-to-end trainable. Experiments on BP4D and DISFA datasets show the effectiveness of our approach. The implementation code is available online.

arXiv comment14 pages,10 figures, published on Neurocomputing

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Action Unit Detection with Region Adaptation, Multi-labeling Learning and Optimal Temporal Fusing

Cited by in corpus (9)

• Facial Action Unit Detection via Adaptive Attention and Relation
• Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
• Contrastive Learning of Person-independent Representations for Facial Action Unit Detection
• Unconstrained Facial Action Unit Detection via Latent Feature Domain
• MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection
• Unobtrusive Pain Monitoring in Older Adults with Dementia using Pairwise and Contrastive Training
• Safety Case Templates for Autonomous Systems
• Computational efficient deep neural network with difference attention maps for facial action unit detection
• Facial Action Unit Detection on ICU Data for Pain Assessment