MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution

Abstract

Facial micro-expression recognition (MER) is a challenging problem, due to transient and subtle micro-expression (ME) actions. Most existing methods depend on hand-crafted features, key frames like onset, apex, and offset frames, or deep networks limited by small-scale and low-diversity datasets. In this paper, we propose an end-to-end micro-action-aware deep learning framework with advantages from transformer, graph convolution, and vanilla convolution. In particular, we propose a novel F5C block composed of fully-connected convolution and channel correspondence convolution to directly extract local-global features from a sequence of raw frames, without the prior knowledge of key frames. The transformer-style fully-connected convolution is proposed to extract local features while maintaining global receptive fields, and the graph-style channel correspondence convolution is introduced to model the correlations among feature patterns. Moreover, MER, optical flow estimation, and facial landmark detection are jointly trained by sharing the local-global features. The two latter tasks contribute to capturing facial subtle action information for MER, which can alleviate the impact of insufficient training data. Extensive experiments demonstrate that our framework (i) outperforms the state-of-the-art MER methods on CASME II, SAMM, and SMIC benchmarks, (ii) works well for optical flow estimation and facial landmark detection, and (iii) can capture facial subtle muscle actions in local regions associated with MEs. The code is available at https://github.com/CYF-cuber/MOL.

This paper has been accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence

References in corpus (10)

• Learning Deep Representation for Face Alignment with Auxiliary Attributes
• Less is More: Micro-expression Recognition from Video using Apex Frame
• Towards Reading Hidden Emotions: A comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods
• Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition
• Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition
• J$\hat{\text{A}}$A-Net: Joint Facial Action Unit Detection and Face Alignment via Adaptive Attention
• LEARNet Dynamic Imaging Network for Micro Expression Recognition
• Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis \& Application
• Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
• Deep Multi-Center Learning for Face Alignment

Cited by in corpus (1)

• CurvNet: Latent Contour Representation and Iterative Data Engine for Curvature Angle Estimation