PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer

Abstract

Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose the PhysFormer, an end-to-end video transformer based architecture, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. Furthermore, we also propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. One highlight is that, unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community. The codes will be released at https://github.com/ZitongYu/PhysFormer.

Accepted by CVPR2022

References in corpus (15)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Is Space-Time Attention All You Need for Video Understanding?
• Transformer in Transformer
• RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
• End-to-end Temporal Action Detection with Transformer
• EVM-CNN: Real-Time Contactless Heart Rate Estimation from Facial Video
• Video Super-Resolution Transformer
• Space-time Mixing Attention for Video Transformer
• Multiscale Vision Transformers
• Video Swin Transformer
• Temporal Action Proposal Generation with Transformers
• EfficientPhys: Enabling Simple, Fast and Accurate Camera-Based Vitals Measurement
• Deep Learning for Face Anti-Spoofing: A Survey