Joint Image-Instance Spatial-Temporal Attention for Few-shot Action Recognition

Abstract

Few-shot Action Recognition (FSAR) constitutes a crucial challenge in computer vision, entailing the recognition of actions from a limited set of examples. Recent approaches mainly focus on employing image-level features to construct temporal dependencies and generate prototypes for each action category. However, a considerable number of these methods utilize mainly image-level features that incorporate background noise and focus insufficiently on real foreground (action-related instances), thereby compromising the recognition capability, particularly in the few-shot scenario. To tackle this issue, we propose a novel joint Image-Instance level Spatial-temporal attention approach (I2ST) for Few-shot Action Recognition. The core concept of I2ST is to perceive the action-related instances and integrate them with image features via spatial-temporal attention. Specifically, I2ST consists of two key components: Action-related Instance Perception and Joint Image-Instance Spatial-temporal Attention. Given the basic representations from the feature extractor, the Action-related Instance Perception is introduced to perceive action-related instances under the guidance of a text-guided segmentation model. Subsequently, the Joint Image-Instance Spatial-temporal Attention is used to construct the feature dependency between instances and images...

Accepted by Computer Vision and Image Understanding

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Prototypical Networks for Few-shot Learning
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Delta-encoder: an effective sample synthesis method for few-shot object recognition
• SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
• Segment Everything Everywhere All at Once
• Few-shot Action Recognition with Prototype-centered Attentive Learning
• Few-Shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning
• Few-shot Action Recognition via Intra- and Inter-Video Information Maximization
• MA-FSAR: Multimodal Adaptation of CLIP for Few-Shot Action Recognition