Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition

Abstract

Humans can easily recognize actions with only a few examples given, while the existing video recognition models still heavily rely on the large-scale labeled data inputs. This observation has motivated an increasing interest in few-shot video action recognition, which aims at learning new actions with only very few labeled samples. In this paper, we propose a depth guided Adaptive Meta-Fusion Network for few-shot video recognition which is termed as AMeFu-Net. Concretely, we tackle the few-shot recognition problem from three aspects: firstly, we alleviate this extremely data-scarce problem by introducing depth information as a carrier of the scene, which will bring extra visual information to our model; secondly, we fuse the representation of original RGB clips with multiple non-strictly corresponding depth clips sampled by our temporal asynchronization augmentation mechanism, which synthesizes new instances at feature-level; thirdly, a novel Depth Guided Adaptive Instance Normalization (DGAdaIN) fusion module is proposed to fuse the two-stream modalities efficiently. Additionally, to better mimic the few-shot recognition process, our model is trained in the meta-learning way. Extensive experiments on several action recognition benchmarks demonstrate the effectiveness of our model.

arXiv commentaccepted by ACM Multimedia 2020

References in corpus (22)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Prototypical Networks for Few-shot Learning
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Multimodal Unsupervised Image-to-Image Translation
• Delta-encoder: an effective sample synthesis method for few-shot object recognition
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition
• Actor-Critic Sequence Training for Image Captioning
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Low-Shot Learning from Imaginary Data
• Learnable Gated Temporal Shift Module for Deep Video Inpainting
• TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
• Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks
• Weakly-supervised Compositional FeatureAggregation for Few-shot Recognition
• Image Deformation Meta-Networks for One-Shot Learning
• Few-shot Action Recognition with Permutation-invariant Attention
• Instance Credibility Inference for Few-Shot Learning
• Long-Term Cloth-Changing Person Re-identification
• Egocentric Action Recognition by Video Attention and Temporal Context
• How to trust unlabeled data? Instance Credibility Inference for Few-Shot Learning