Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos

Abstract

Deep learning has been demonstrated to achieve excellent results for image classification and object detection. However, the impact of deep learning on video analysis (e.g. action detection and recognition) has been limited due to complexity of video data and lack of annotations. Previous convolutional neural networks (CNN) based video action detection approaches usually consist of two major steps: frame-level action proposal detection and association of proposals across frames. Also, these methods employ two-stream CNN framework to handle spatial and temporal feature separately. In this paper, we propose an end-to-end deep network called Tube Convolutional Neural Network (T-CNN) for action detection in videos. The proposed architecture is a unified network that is able to recognize and localize action based on 3D convolution features. A video is first divided into equal length clips and for each clip a set of tube proposals are generated next based on 3D Convolutional Network (ConvNet) features. Finally, the tube proposals of different clips are linked together employing network flow and spatio-temporal action detection is performed using these linked video proposals. Extensive experiments on several video datasets demonstrate the superior performance of T-CNN for classifying and localizing actions in both trimmed and untrimmed videos compared to state-of-the-arts.

References in corpus (3)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Caffe: Convolutional Architecture for Fast Feature Embedding
• YOLO9000: Better, Faster, Stronger

Cited by in corpus (16)

• Analyzing Human-Human Interactions: A Survey
• A Better Baseline for AVA
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• VideoCapsuleNet: A Simplified Network for Action Detection
• Audio-only Bird Species Automated Identification Method with Limited Training Data Based on Multi-Channel Deep Convolutional Neural Networks
• Spatio-temporal Action Recognition: A Survey
• Real-Time End-to-End Action Detection with Two-Stream Networks
• Multi-modal Capsule Routing for Actor and Action Video Segmentation Conditioned on Natural Language Queries
• ReHAR: Robust and Efficient Human Activity Recognition
• Spatio-Temporal Instance Learning: Action Tubes from Class Supervision
• Context-Aware RCNN: A Baseline for Action Detection in Videos
• Multilevel Language and Vision Integration for Text-to-Clip Retrieval
• Action Detection from a Robot-Car Perspective
• Time and Frequency Network for Human Action Detection in Videos
• Long Short-Term Relation Networks for Video Action Detection
• Discovering Spatio-Temporal Action Tubes