PGT: A Progressive Method for Training Models on Long Videos

Abstract

Convolutional video models have an order of magnitude larger computational complexity than their counterpart image-level models. Constrained by computational resources, there is no model or training method that can train long video sequences end-to-end. Currently, the main-stream method is to split a raw video into clips, leading to incomplete fragmentary temporal information flow. Inspired by natural language processing techniques dealing with long sentences, we propose to treat videos as serial fragments satisfying Markov property, and train it as a whole by progressively propagating information through the temporal dimension in multiple steps. This progressive training (PGT) method is able to train long videos end-to-end with limited resources and ensures the effective transmission of information. As a general and robust training method, we empirically demonstrate that it yields significant performance improvements on different models and datasets. As an illustrative example, the proposed method improves SlowOnly network by 3.7 mAP on Charades and 1.9 top-1 accuracy on Kinetics with negligible parameter and computation overhead. Code is available at https://github.com/BoPang1996/PGT.

arXiv commentCVPR21, Oral

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• A Short Note about Kinetics-600
• Learnable pooling with Context Gating for video classification
• Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding
• VideoGraph: Recognizing Minutes-Long Human Activities in Videos
• Complex Sequential Understanding through the Awareness of Spatial and Temporal Concepts
• PIC: Permutation Invariant Convolution for Recognizing Long-range Activities
• TDAF: Top-Down Attention Framework for Vision Tasks