Temporal Segment Networks for Action Recognition in Videos

Abstract

Deep convolutional networks have achieved great success for image recognition. However, for action recognition in videos, their advantage over traditional methods is not so evident. We present a general and flexible video-level framework for learning action models in videos. This method, called temporal segment network (TSN), aims to model long-range temporal structures with a new segment-based sampling and aggregation module. This unique design enables our TSN to efficiently learn action models by using the whole action videos. The learned models could be easily adapted for action recognition in both trimmed and untrimmed videos with simple average pooling and multi-scale temporal window integration, respectively. We also study a series of good practices for the instantiation of TSN framework given limited training samples. Our approach obtains the state-the-of-art performance on four challenging action recognition benchmarks: HMDB51 (71.0%), UCF101 (94.9%), THUMOS14 (80.1%), and ActivityNet v1.2 (89.6%). Using the proposed RGB difference for motion models, our method can still achieve competitive accuracy on UCF101 (91.0%) while running at 340 FPS. Furthermore, based on the temporal segment networks, we won the video classification track at the ActivityNet challenge 2016 among 24 teams, which demonstrates the effectiveness of TSN and the proposed good practices.

arXiv comment14 pages. An extension of submission at https://arxiv.org/abs/1608.00859

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Spatiotemporal Residual Networks for Video Action Recognition
• Towards Good Practices for Very Deep Two-Stream ConvNets
• Long-term Temporal Convolutions for Action Recognition
• Efficient Two-Stream Motion and Appearance 3D CNNs for Video Classification
• Learning Multiscale Features Directly From Waveforms

Cited by in corpus (6)

• Step-by-step Erasion, One-by-one Collection: A Weakly Supervised Temporal Action Detector
• End-to-end Video-level Representation Learning for Action Recognition
• HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognition
• Universal-to-Specific Framework for Complex Action Recognition
• Qiniu Submission to ActivityNet Challenge 2018
• Temporal Accumulative Features for Sign Language Recognition