Residual Frames with Efficient Pseudo-3D CNN for Human Action Recognition

Abstract

Human action recognition is regarded as a key cornerstone in domains such as surveillance or video understanding. Despite recent progress in the development of end-to-end solutions for video-based action recognition, achieving state-of-the-art performance still requires using auxiliary hand-crafted motion representations, e.g., optical flow, which are usually computationally demanding. In this work, we propose to use residual frames (i.e., differences between adjacent RGB frames) as an alternative "lightweight" motion representation, which carries salient motion information and is computationally efficient. In addition, we develop a new pseudo-3D convolution module which decouples 3D convolution into 2D and 1D convolution. The proposed module exploits residual information in the feature space to better structure motions, and is equipped with a self-attention mechanism that assists to recalibrate the appearance and motion features. Empirical results confirm the efficiency and effectiveness of residual frames as well as the proposed pseudo-3D convolution module.

References in corpus (6)

• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• SlowFast Networks for Video Recognition
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Semi-Coupled Two-Stream Fusion ConvNets for Action Recognition at Extremely Low Resolutions