Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition

Abstract

Being spontaneous, micro-expressions are useful in the inference of a person's true emotions even if an attempt is made to conceal them. Due to their short duration and low intensity, the recognition of micro-expressions is a difficult task in affective computing. The early work based on handcrafted spatio-temporal features which showed some promise, has recently been superseded by different deep learning approaches which now compete for the state of the art performance. Nevertheless, the problem of capturing both local and global spatio-temporal patterns remains challenging. To this end, herein we propose a novel spatio-temporal transformer architecture -- to the best of our knowledge, the first purely transformer based approach (i.e. void of any convolutional network use) for micro-expression recognition. The architecture comprises a spatial encoder which learns spatial patterns, a temporal aggregator for temporal dimension analysis, and a classification head. A comprehensive evaluation on three widely used spontaneous micro-expression data sets, namely SMIC-HS, CASME II and SAMM, shows that the proposed approach consistently outperforms the state of the art, and is the first framework in the published literature on micro-expression recognition to achieve the unweighted F1-score greater than 0.9 on any of the aforementioned data sets.

arXiv comment13 pages, 9 figures

References in corpus (8)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Transformers in Vision: A Survey
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Less is More: Micro-expression Recognition from Video using Apex Frame
• Towards Reading Hidden Emotions: A comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods
• Facial Expression Recognition with Visual Transformers and Attentional Selective Fusion
• Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
• Revealing the Invisible with Model and Data Shrinking for Composite-database Micro-expression Recognition

Cited by in corpus (6)

• Benchmarking Micro-action Recognition: Dataset, Methods, and Applications
• MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
• Data Leakage and Evaluation Issues in Micro-Expression Analysis
• MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection
• Micro-Expression Recognition via Fine-Grained Dynamic Perception
• Navigating human-robot interaction challenges in private and residential aged care homes