Learn to cycle: Time-consistent feature discovery for action recognition

Abstract

Generalizing over temporal variations is a prerequisite for effective action recognition in videos. Despite significant advances in deep neural networks, it remains a challenge to focus on short-term discriminative motions in relation to the overall performance of an action. We address this challenge by allowing some flexibility in discovering relevant spatio-temporal features. We introduce Squeeze and Recursion Temporal Gates (SRTG), an approach that favors inputs with similar activations with potential temporal variations. We implement this idea with a novel CNN block that uses an LSTM to encapsulate feature dynamics, in conjunction with a temporal gate that is responsible for evaluating the consistency of the discovered dynamics and the modeled features. We show consistent improvement when using SRTG blocks, with only a minimal increase in the number of GFLOPs. On Kinetics-700, we perform on par with current state-of-the-art models, and outperform these on HACS, Moments in Time, UCF-101 and HMDB-51.

References in corpus (7)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• Video Classification with Channel-Separated Convolutional Networks
• Analyzing Human-Human Interactions: A Survey
• Spatio-Temporal FAST 3D Convolutions for Human Action Recognition

Cited by in corpus (5)

• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• Learning from Videos with Deep Convolutional LSTM Networks
• Learning Class Regularized Features for Action Recognition
• Efficient Modelling Across Time of Human Actions and Interactions