Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding

Abstract

This paper describes our solution for the video recognition task of the Google Cloud and YouTube-8M Video Understanding Challenge that ranked the 3rd place. Because the challenge provides pre-extracted visual and audio features instead of the raw videos, we mainly investigate various temporal modeling approaches to aggregate the frame-level features for multi-label video recognition. Our system contains three major components: two-stream sequence model, fast-forward sequence model and temporal residual neural networks. Experiment results on the challenging Youtube-8M dataset demonstrate that our proposed temporal modeling approaches can significantly improve existing temporal modeling approaches in the large-scale video recognition tasks. To be noted, our fast-forward LSTM with a depth of 7 layers achieves 82.75% in term of GAP@20 on the Kaggle Public test set.

arXiv commentTo appear on CVPR 2017 YouTube-8M Workshop(Rank 3rd out of 650 teams)

References in corpus (4)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Unsupervised Learning of Video Representations using LSTMs
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Deep Speaker: an End-to-End Neural Speaker Embedding System

Cited by in corpus (17)

• Learnable pooling with Context Gating for video classification
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• Long-Term Feature Banks for Detailed Video Understanding
• From Deterministic to Generative: Multi-Modal Stochastic RNNs for Video Captioning
• Non-local NetVLAD Encoding for Video Classification
• Learning from Label Relationships in Human Affect
• Efficient Video Classification Using Fewer Frames
• Temporal Query Networks for Fine-grained Video Understanding
• Learning Efficient Video Representation with Video Shuffle Networks
• BERT for Large-scale Video Segment Classification with Test-time Augmentation
• I Have Seen Enough: A Teacher Student Network for Video Classification Using Fewer Frames
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• PGT: A Progressive Method for Training Models on Long Videos
• Multi-attention Networks for Temporal Localization of Video-level Labels
• Deep Multimodal Learning: An Effective Method for Video Classification
• Training compact deep learning models for video classification using circulant matrices
• Constrained-size Tensorflow Models for YouTube-8M Video Understanding Challenge