Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
arXiv:2006.07976
Abstract
Localizing persons and recognizing their actions from videos is a challenging task towards high-level video understanding. Recent advances have been achieved by modeling direct pairwise relations between entities. In this paper, we take one step further, not only model direct relations between pairs but also take into account indirect higher-order relations established upon multiple elements. We propose to explicitly model the Actor-Context-Actor Relation, which is the relation between two actors based on their interactions with the context. To this end, we design an Actor-Context-Actor Relation Network (ACAR-Net) which builds upon a novel High-order Relation Reasoning Operator and an Actor-Context Feature Bank to enable indirect relation reasoning for spatio-temporal action localization. Experiments on AVA and UCF101-24 datasets show the advantages of modeling actor-context-actor relations, and visualization of attention maps further verifies that our model is capable of finding relevant higher-order relations to support action detection. Notably, our method ranks first in the AVA-Kineticsaction localization task of ActivityNet Challenge 2020, out-performing other entries by a significant margin (+6.71mAP). Training code and models will be available at https://github.com/Siyu-C/ACAR-Net.
arXiv commentAccepted in CVPR 2021
References in corpus (15)
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- Audiovisual SlowFast Networks for Video Recognition
- You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization
- The AVA-Kinetics Localized Human Actions Video Dataset
- A Better Baseline for AVA
- Temporal Relational Reasoning in Videos
- Deep Contextual Attention for Human-Object Interaction Detection
- Video action detection by learning graph-based spatio-temporal interactions
- Asynchronous Interaction Aggregation for Action Detection
- Three Branches: Detecting Actions With Richer Features
- 1st place solution for AVA-Kinetics Crossover in AcitivityNet Challenge 2020
- Context-Aware RCNN: A Baseline for Action Detection in Videos