Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition

Abstract

Convolutional neural networks with spatio-temporal 3D kernels (3D CNNs) have an ability to directly extract spatio-temporal features from videos for action recognition. Although the 3D kernels tend to overfit because of a large number of their parameters, the 3D CNNs are greatly improved by using recent huge video databases. However, the architecture of 3D CNNs is relatively shallow against to the success of very deep neural networks in 2D-based CNNs, such as residual networks (ResNets). In this paper, we propose a 3D CNNs based on ResNets toward a better action representation. We describe the training procedure of our 3D ResNets in details. We experimentally evaluate the 3D ResNets on the ActivityNet and Kinetics datasets. The 3D ResNets trained on the Kinetics did not suffer from overfitting despite the large number of parameters of the model, and achieved better performance than relatively shallow networks, such as C3D. Our code and pretrained models (e.g. Kinetics and ActivityNet) are publicly available at https://github.com/kenshohara/3D-ResNets.

To appear in ICCV 2017 Workshop (Chalearn)

References in corpus (8)

• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Densely Connected Convolutional Networks
• Going Deeper with Convolutions
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Learning Spatiotemporal Features with 3D Convolutional Networks

Cited by in corpus (11)

• Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification
• An Attempt towards Interpretable Audio-Visual Video Captioning
• Unsupervised Learning from Video with Deep Neural Embeddings
• Deep Face Forgery Detection
• Motion Feature Network: Fixed Motion Filter for Action Recognition
• The eyes know it: FakeET -- An Eye-tracking Database to Understand Deepfake Perception
• Drive Video Analysis for the Detection of Traffic Near-Miss Incidents
• Fine-grained Video Categorization with Redundancy Reduction Attention
• T-RECS: Training for Rate-Invariant Embeddings by Controlling Speed for Action Recognition
• Perceptron Synthesis Network: Rethinking the Action Scale Variances in Videos
• Simulating time to event prediction with spatiotemporal echocardiography deep learning