Less is More: Micro-expression Recognition from Video using Apex Frame

Abstract

Despite recent interest and advances in facial micro-expression research, there is still plenty room for improvement in terms of micro-expression recognition. Conventional feature extraction approaches for micro-expression video consider either the whole video sequence or a part of it, for representation. However, with the high-speed video capture of micro-expressions (100-200 fps), are all frames necessary to provide a sufficiently meaningful representation? Is the luxury of data a bane to accurate recognition? A novel proposition is presented in this paper, whereby we utilize only two images per video: the apex frame and the onset frame. The apex frame of a video contains the highest intensity of expression changes among all frames, while the onset is the perfect choice of a reference frame with neutral expression. A new feature extractor, Bi-Weighted Oriented Optical Flow (Bi-WOOF) is proposed to encode essential expressiveness of the apex frame. We evaluated the proposed method on five micro-expression databases: CAS(ME)${}^2$, CASME II, SMIC-HS, SMIC-NIR and SMIC-VIS. Our experiments lend credence to our hypothesis, with our proposed technique achieving a state-of-the-art F1-score recognition performance of 61% and 62% in the high frame rate CASME II and SMIC-HS databases respectively.

14 pages double-column, author affiliations updated, acknowledgment of grant support added

References in corpus (2)

• Sparsity in Dynamics of Spontaneous Subtle Emotions: Analysis \& Application
• Spontaneous Subtle Expression Detection and Recognition based on Facial Strain

Cited by in corpus (20)

• Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition
• Revealing the Invisible with Model and Data Shrinking for Composite-database Micro-expression Recognition
• A Survey of Automatic Facial Micro-expression Analysis: Databases, Methods and Challenges
• Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition
• A Review on Facial Micro-Expressions Analysis: Datasets, Features and Metrics
• DFME: A New Benchmark for Dynamic Facial Micro-expression Recognition
• MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
• Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition
• Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey
• Micro-Attention for Micro-Expression recognition
• MER-GCN: Micro Expression Recognition Based on Relation Modeling with Graph Convolutional Network
• A Novel Apex-Time Network for Cross-Dataset Micro-Expression Recognition
• Recognizing Micro-Expression in Video Clip with Adaptive Key-Frame Mining
• Evaluation of the Spatio-Temporal features and GAN for Micro-expression Recognition System
• Micro-Expression Recognition via Fine-Grained Dynamic Perception
• Automatic Micro-Expression Apex Frame Spotting using Local Binary Pattern from Six Intersection Planes
• Automatic Surface Area and Volume Prediction on Ellipsoidal Ham using Deep Learning
• Region attention and graph embedding network for occlusion objective class-based micro-expression recognition
• A Dynamic 3D Spontaneous Micro-expression Database: Establishment and Evaluation
• A comparative study on movement feature in different directions for micro-expression recognition