Real-time Action Recognition with Enhanced Motion Vector CNNs

Abstract

The deep two-stream architecture exhibited excellent performance on video based action recognition. The most computationally expensive step in this approach comes from the calculation of optical flow which prevents it to be real-time. This paper accelerates this architecture by replacing optical flow with motion vector which can be obtained directly from compressed videos without extra calculation. However, motion vector lacks fine structures, and contains noisy and inaccurate motion patterns, leading to the evident degradation of recognition performance. Our key insight for relieving this problem is that optical flow and motion vector are inherent correlated. Transferring the knowledge learned with optical flow CNN to motion vector CNN can significantly boost the performance of the latter. Specifically, we introduce three strategies for this, initialization transfer, supervision transfer and their combination. Experimental results show that our method achieves comparable recognition performance to the state-of-the-art, while our method can process 390.7 frames per second, which is 27 times faster than the original two-stream method.

arXiv commentaccepted by CVPR16

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Towards Good Practices for Very Deep Two-Stream ConvNets
• Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice

Cited by in corpus (19)

• CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
• Hidden Two-Stream Convolutional Networks for Action Recognition
• V4D:4D Convolutional Neural Networks for Video-level Representation Learning
• Pedestrian Attribute Recognition: A Survey
• Memory Attention Networks for Skeleton-based Action Recognition
• Lattice Long Short-Term Memory for Human Action Recognition
• Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification
• Deep Action- and Context-Aware Sequence Learning for Activity Recognition and Anticipation
• Euphrates: Algorithm-SoC Co-Design for Low-Power Mobile Continuous Vision
• DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition
• Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classification
• Learning to score the figure skating sports videos
• Eigen Evolution Pooling for Human Action Recognition
• Generalized Zero-Shot Learning for Action Recognition with Web-Scale Video Data
• Self-Paced Video Data Augmentation with Dynamic Images Generated by Generative Adversarial Networks
• TEAM-Net: Multi-modal Learning for Video Action Recognition with Partial Decoding
• Motion Feature Network: Fixed Motion Filter for Action Recognition
• Inter-BMV: Interpolation with Block Motion Vectors for Fast Semantic Segmentation on Video
• Self-Referenced Deep Learning