A${}^2$M${}^2$-Net: Adaptively Aligned Multi-Scale Moment for Few-Shot Action Recognition

Abstract

Thanks to capability to alleviate the cost of large-scale annotation, few-shot action recognition (FSAR) has attracted increased attention of researchers in recent years. Existing FSAR approaches typically neglect the role of individual motion pattern in comparison, and under-explore the feature statistics for video dynamics. Thereby, they struggle to handle the challenging temporal misalignment in video dynamics, particularly by using 2D backbones. To overcome these limitations, this work proposes an adaptively aligned multi-scale second-order moment network, namely A${}^2$M${}^2$-Net, to describe the latent video dynamics with a collection of powerful representation candidates and adaptively align them in an instance-guided manner. To this end, our A${}^2$M${}^2$-Net involves two core components, namely, adaptive alignment (A${}^2$ module) for matching, and multi-scale second-order moment (M${}^2$ block) for strong representation. Specifically, M${}^2$ block develops a collection of semantic second-order descriptors at multiple spatio-temporal scales. Furthermore, A${}^2$ module aims to adaptively select informative candidate descriptors while considering the individual motion pattern. By such means, our A${}^2$M${}^2$-Net is able to handle the challenging temporal misalignment problem by establishing an adaptive alignment protocol for strong representation. Notably, our proposed method generalizes well to various few-shot settings and diverse metrics. The experiments are conducted on five widely used FSAR benchmarks, and the results show our A${}^2$M${}^2$-Net achieves very competitive performance compared to state-of-the-arts, demonstrating its effectiveness and generalization.

27 pages, 13 figures, 7 tables

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Learning to learn by gradient descent by gradient descent
• Deep CNNs Meet Global Covariance Pooling: Better Representation and Generalization
• TA2N: Two-Stage Action Alignment Network for Few-shot Action Recognition
• CrossTransformers: spatially-aware few-shot transfer
• DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning
• TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
• Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers
• Few-Shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning
• Temporal-attentive Covariance Pooling Networks for Video Recognition
• Few-shot Action Recognition with Implicit Temporal Alignment and Pair Similarity Optimization