ConvNet Architecture Search for Spatiotemporal Feature Learning

Abstract

Learning image representations with ConvNets by pre-training on ImageNet has proven useful across many visual understanding tasks including object detection, semantic segmentation, and image captioning. Although any image representation can be applied to video frames, a dedicated spatiotemporal representation is still vital in order to incorporate motion patterns that cannot be captured by appearance based models alone. This paper presents an empirical ConvNet architecture search for spatiotemporal feature learning, culminating in a deep 3-dimensional (3D) Residual ConvNet. Our proposed architecture outperforms C3D by a good margin on Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN while being 2 times faster at inference time, 2 times smaller in model size, and having a more compact representation.

Cited by in corpus (54)

• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
• Dynamic Sampling Networks for Efficient Action Recognition in Videos
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?
• BMN: Boundary-Matching Network for Temporal Action Proposal Generation
• A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data Modalities
• GL-RG: Global-Local Representation Granularity for Video Captioning
• Appearance-and-Relation Networks for Video Classification
• Aligning Correlation Information for Domain Adaptation in Action Recognition
• Long-Term Feature Banks for Detailed Video Understanding
• PAN: Towards Fast Action Recognition via Learning Persistence of Appearance
• Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
• Multi-Fiber Networks for Video Recognition
• $A^2$-Nets: Double Attention Networks
• Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks
• Mutual Suppression Network for Video Prediction using Disentangled Features
• DMC-Net: Generating Discriminative Motion Cues for Fast Compressed Video Action Recognition
• Learning Video Representations from Correspondence Proposals
• Temporal Gaussian Mixture Layer for Videos
• Learning from Temporal Gradient for Semi-supervised Action Recognition
• PERF-Net: Pose Empowered RGB-Flow Net
• Towards Real-Time Action Recognition on Mobile Devices Using Deep Models
• Flow-Distilled IP Two-Stream Networks for Compressed Video Action Recognition
• Compressed Video Action Recognition with Refined Motion Vector
• IF-TTN: Information Fused Temporal Transformation Network for Video Action Recognition
• Dynamic Temporal Pyramid Network: A Closer Look at Multi-Scale Modeling for Activity Detection
• Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition
• LP-3DCNN: Unveiling Local Phase in 3D Convolutional Neural Networks
• RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks
• Cooperative Cross-Stream Network for Discriminative Action Representation
• 3DPalsyNet: A Facial Palsy Grading and Motion Recognition Framework using Fully 3D Convolutional Neural Networks
• Towards Train-Test Consistency for Semi-supervised Temporal Action Localization
• AVD: Adversarial Video Distillation
• Motion Feature Network: Fixed Motion Filter for Action Recognition
• Temporal Reasoning Graph for Activity Recognition
• Dynamic Graph Modules for Modeling Object-Object Interactions in Activity Recognition
• Multimodal Explanations by Predicting Counterfactuality in Videos
• Temporal Bilinear Networks for Video Action Recognition
• Action Search: Spotting Actions in Videos and Its Application to Temporal Action Localization
• AutoLoc: Weakly-supervised Temporal Action Localization
• Spatio-Temporal Channel Correlation Networks for Action Classification
• Exploiting Inter-Frame Regional Correlation for Efficient Action Recognition
• Temporal Bilinear Encoding Network of Audio-Visual Features at Low Sampling Rates
• Efficient Spatialtemporal Context Modeling for Action Recognition
• M-PACT: An Open Source Platform for Repeatable Activity Classification Research
• Online Detection of Action Start in Untrimmed, Streaming Videos
• Automated identification of neural cells in the multi-photon images using deep-neural networks
• Deep set conditioned latent representations for action recognition
• Learning to Sort Image Sequences via Accumulated Temporal Differences
• Approximated Bilinear Modules for Temporal Modeling
• A Real-time Action Representation with Temporal Encoding and Deep Compression
• Channel-Temporal Attention for First-Person Video Domain Adaptation