Beyond Short Snippets: Deep Networks for Video Classification

Abstract

Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 72.8%).

References in corpus (4)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Going Deeper with Convolutions
• Learning to Execute
• Phoneme recognition in TIMIT with BLSTM-CTC

Cited by in corpus (27)

• Learning to Diagnose with LSTM Recurrent Neural Networks
• Delving Deeper into Convolutional Networks for Learning Video Representations
• Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks
• Exploiting deep residual networks for human action recognition from skeletal data
• Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
• Improving performance of recurrent neural network with relu nonlinearity
• Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading
• Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification
• End-to-end Learning of Action Detection from Frame Glimpses in Videos
• Deep Unsupervised Learning of Visual Similarities
• Connectionist Temporal Modeling for Weakly Supervised Action Labeling
• Complex Sequential Understanding through the Awareness of Spatial and Temporal Concepts
• Efficient Large Scale Video Classification
• Clean-Label Backdoor Attacks on Video Recognition Models
• Spatio-Temporal Fusion Networks for Action Recognition
• IterGANs: Iterative GANs to Learn and Control 3D Object Transformation
• Handcrafted Local Features are Convolutional Neural Networks
• Learning to score the figure skating sports videos
• Learning Fine-grained Features via a CNN Tree for Large-scale Classification
• Object Recognition from Short Videos for Robotic Perception
• Recurrent Neural Networks for P300-based BCI
• The Best of Both Worlds: Combining Data-independent and Data-driven Approaches for Action Recognition
• Large-Scale YouTube-8M Video Understanding with Deep Neural Networks
• Exploring Uncertainty Measures for Image-Caption Embedding-and-Retrieval Task
• Do Less and Achieve More: Training CNNs for Action Recognition Utilizing Action Images from the Web
• Large age-gap face verification by feature injection in deep networks
• Spatio-Temporal Deep Learning Models for Tip Force Estimation During Needle Insertion