Towards Good Practices for Very Deep Two-Stream ConvNets

Abstract

Deep convolutional networks have achieved great success for object recognition in still images. However, for action recognition in videos, the improvement of deep convolutional networks is not so evident. We argue that there are two reasons that could probably explain this result. First the current network architectures (e.g. Two-stream ConvNets) are relatively shallow compared with those very deep models in image domain (e.g. VGGNet, GoogLeNet), and therefore their modeling capacity is constrained by their depth. Second, probably more importantly, the training dataset of action recognition is extremely small compared with the ImageNet dataset, and thus it will be easy to over-fit on the training dataset. To address these issues, this report presents very deep two-stream ConvNets for action recognition, by adapting recent very deep architectures into video domain. However, this extension is not easy as the size of action recognition is quite small. We design several good practices for the training of very deep two-stream ConvNets, namely (i) pre-training for both spatial and temporal nets, (ii) smaller learning rates, (iii) more data augmentation techniques, (iv) high drop out ratio. Meanwhile, we extend the Caffe toolbox into Multi-GPU implementation with high computational efficiency and low memory consumption. We verify the performance of very deep two-stream ConvNets on the dataset of UCF101 and it achieves the recognition accuracy of $91.4\%$.

References in corpus (5)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Going Deeper with Convolutions
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice

Cited by in corpus (81)

• Human Action Recognition from Various Data Modalities: A Review
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Pyramid Scene Parsing Network
• Revisiting Anchor Mechanisms for Temporal Action Localization
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Delving Deeper into Convolutional Networks for Learning Video Representations
• Long-term Temporal Convolutions for Action Recognition
• Places205-VGGNet Models for Scene Recognition
• A Comprehensive Study of Deep Video Action Recognition
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• Hidden Two-Stream Convolutional Networks for Action Recognition
• BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?
• Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks
• BMN: Boundary-Matching Network for Temporal Action Proposal Generation
• Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification
• Real-time Action Recognition with Enhanced Motion Vector CNNs
• Temporal Segment Networks for Action Recognition in Videos
• VideoMix: Rethinking Data Augmentation for Video Classification
• Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification
• S3D: Single Shot multi-Span Detector via Fully 3D Convolutional Networks
• Predictive-Corrective Networks for Action Detection
• MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
• Efficient Two-Stream Motion and Appearance 3D CNNs for Video Classification
• Unsupervised Learning of Long-Term Motion Dynamics for Videos
• Correlation Net: Spatiotemporal multimodal deep learning for action recognition
• Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
• G-TAD: Sub-Graph Localization for Temporal Action Detection
• Video-based Human Action Recognition using Deep Learning: A Review
• Actionness Estimation Using Hybrid Fully Convolutional Networks
• Taming Visually Guided Sound Generation
• Depth2Action: Exploring Embedded Depth for Large-Scale Action Recognition
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• Fast Learning of Temporal Action Proposal via Dense Boundary Generator
• Let's Dance: Learning From Online Dance Videos
• Temporal Context Aggregation Network for Temporal Action Proposal Refinement
• YH Technologies at ActivityNet Challenge 2018
• Zero-Shot Visual Recognition via Bidirectional Latent Embedding
• Learning from Temporal Gradient for Semi-supervised Action Recognition
• Boundary Content Graph Neural Network for Temporal Action Proposal Generation
• Pooling the Convolutional Layers in Deep ConvNets for Action Recognition
• UC Merced Submission to the ActivityNet Challenge 2016
• Spatio-temporal Person Retrieval via Natural Language Queries
• Body Joint guided 3D Deep Convolutional Descriptors for Action Recognition
• Factors in Finetuning Deep Model for object detection
• Weakly supervised learning of actions from transcripts
• Learning long-term dependencies for action recognition with a biologically-inspired deep network
• Learning Discriminative Features via Label Consistent Neural Network
• RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks
• Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks
• Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction
• ActionVLAD: Learning spatio-temporal aggregation for action classification
• Disjoint Multi-task Learning between Heterogeneous Human-centric Tasks
• Video Representation Learning and Latent Concept Mining for Large-scale Multi-label Video Classification
• Dance with Flow: Two-in-One Stream Action Detection
• P-ODN: Prototype based Open Deep Network for Open Set Recognition
• TTPP: Temporal Transformer with Progressive Prediction for Efficient Action Anticipation
• Random Temporal Skipping for Multirate Video Analysis
• Video Modeling with Correlation Networks
• Temporal Action Localization by Structured Maximal Sums
• A 3D CNN Network with BERT For Automatic COVID-19 Diagnosis From CT-Scan Images
• Transferable Knowledge-Based Multi-Granularity Aggregation Network for Temporal Action Localization: Submission to ActivityNet Challenge 2021
• Efficient Action Detection in Untrimmed Videos via Multi-Task Learning
• Improved Dense Trajectory with Cross Streams
• 2nd Place Scheme on Action Recognition Track of ECCV 2020 VIPriors Challenges: An Efficient Optical Flow Stream Guided Framework
• Adversarial Learning of Label Dependency: A Novel Framework for Multi-class Classification
• Better Exploiting OS-CNNs for Better Event Recognition in Images
• Deep set conditioned latent representations for action recognition
• A Comprehensive Study on Temporal Modeling for Online Action Detection
• AdaScan: Adaptive Scan Pooling in Deep Convolutional Neural Networks for Human Action Recognition in Videos
• Recurrent Deconvolutional Generative Adversarial Networks with Application to Text Guided Video Generation
• Temporal Action Detection by Joint Identification-Verification
• An Order Preserving Bilinear Model for Person Detection in Multi-Modal Data
• Attentive Sequence to Sequence Translation for Localizing Clips of Interest by Natural Language Descriptions
• Exploring Temporal Information for Improved Video Understanding
• Fusing Deep Convolutional Networks for Large Scale Visual Concept Classification
• Deep Point-wise Prediction for Action Temporal Proposal
• Attention Filtering for Multi-person Spatiotemporal Action Detection on Deep Two-Stream CNN Architectures
• Reconstruction Regularized Deep Metric Learning for Multi-label Image Classification
• Large-Scale Mapping of Human Activity using Geo-Tagged Videos
• Action Recognition Using Volumetric Motion Representations