Towards Reading Hidden Emotions: A comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods

Abstract

Micro-expressions (MEs) are rapid, involuntary facial expressions which reveal emotions that people do not intend to show. Studying MEs is valuable as recognizing them has many important applications, particularly in forensic science and psychotherapy. However, analyzing spontaneous MEs is very challenging due to their short duration and low intensity. Automatic ME analysis includes two tasks: ME spotting and ME recognition. For ME spotting, previous studies have focused on posed rather than spontaneous videos. For ME recognition, the performance of previous studies is low. To address these challenges, we make the following contributions: (i)We propose the first method for spotting spontaneous MEs in long videos (by exploiting feature difference contrast). This method is training free and works on arbitrary unseen videos. (ii)We present an advanced ME recognition framework, which outperforms previous work by a large margin on two challenging spontaneous ME databases (SMIC and CASMEII). (iii)We propose the first automatic ME analysis system (MESR), which can spot and recognize MEs from spontaneous video data. Finally, we show our method outperforms humans in the ME recognition task by a large margin, and achieves comparable performance to humans at the very challenging task of spotting and then recognizing spontaneous MEs.

Cited by in corpus (13)

• Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms
• Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition
• Revealing the Invisible with Model and Data Shrinking for Composite-database Micro-expression Recognition
• A Survey of Automatic Facial Micro-expression Analysis: Databases, Methods and Challenges
• Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• Learning a Target Sample Re-Generator for Cross-Database Micro-Expression Recognition
• DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition
• MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
• Data Leakage and Evaluation Issues in Micro-Expression Analysis
• Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey
• Micro-Expression Recognition via Fine-Grained Dynamic Perception
• Causal-Ex: Causal Graph-based Micro and Macro Expression Spotting