Towards Good Practices for Very Deep Two-Stream ConvNets

Abstract

Deep convolutional networks have achieved great success for object recognition in still images. However, for action recognition in videos, the improvement of deep convolutional networks is not so evident. We argue that there are two reasons that could probably explain this result. First the current network architectures (e.g. Two-stream ConvNets) are relatively shallow compared with those very deep models in image domain (e.g. VGGNet, GoogLeNet), and therefore their modeling capacity is constrained by their depth. Second, probably more importantly, the training dataset of action recognition is extremely small compared with the ImageNet dataset, and thus it will be easy to over-fit on the training dataset. To address these issues, this report presents very deep two-stream ConvNets for action recognition, by adapting recent very deep architectures into video domain. However, this extension is not easy as the size of action recognition is quite small. We design several good practices for the training of very deep two-stream ConvNets, namely (i) pre-training for both spatial and temporal nets, (ii) smaller learning rates, (iii) more data augmentation techniques, (iv) high drop out ratio. Meanwhile, we extend the Caffe toolbox into Multi-GPU implementation with high computational efficiency and low memory consumption. We verify the performance of very deep two-stream ConvNets on the dataset of UCF101 and it achieves the recognition accuracy of $91.4\%$.

References in corpus (4)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Going Deeper with Convolutions
• Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice

Cited by in corpus (14)

• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
• BMN: Boundary-Matching Network for Temporal Action Proposal Generation
• Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification
• Efficient Two-Stream Motion and Appearance 3D CNNs for Video Classification
• Depth2Action: Exploring Embedded Depth for Large-Scale Action Recognition
• Let's Dance: Learning From Online Dance Videos
• Learning long-term dependencies for action recognition with a biologically-inspired deep network
• Dance with Flow: Two-in-One Stream Action Detection
• Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
• Large-Scale Mapping of Human Activity using Geo-Tagged Videos
• An Order Preserving Bilinear Model for Person Detection in Multi-Modal Data
• Fusing Deep Convolutional Networks for Large Scale Visual Concept Classification
• Exploring Temporal Information for Improved Video Understanding