A Structured Model For Action Detection

Abstract

A dominant paradigm for learning-based approaches in computer vision is training generic models, such as ResNet for image recognition, or I3D for video understanding, on large datasets and allowing them to discover the optimal representation for the problem at hand. While this is an obviously attractive approach, it is not applicable in all scenarios. We claim that action detection is one such challenging problem - the models that need to be trained are large, and labeled data is expensive to obtain. To address this limitation, we propose to incorporate domain knowledge into the structure of the model, simplifying optimization. In particular, we augment a standard I3D network with a tracking module to aggregate long term motion patterns, and use a graph convolutional network to reason about interactions between actors and objects. Evaluated on the challenging AVA dataset, the proposed approach improves over the I3D baseline by 5.5% mAP and over the state-of-the-art by 4.8% mAP.

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• In Defense of the Triplet Loss for Person Re-Identification
• Online Tracking by Learning Discriminative Saliency Map with Convolutional Neural Network
• Visual Semantic Role Labeling
• Object Level Visual Reasoning in Videos