Low-Fidelity End-to-End Video Encoder Pre-training for Temporal Action Localization

Abstract

Temporal action localization (TAL) is a fundamental yet challenging task in video understanding. Existing TAL methods rely on pre-training a video encoder through action classification supervision. This results in a task discrepancy problem for the video encoder -- trained for action classification, but used for TAL. Intuitively, end-to-end model optimization is a good solution. However, this is not operable for TAL subject to the GPU memory constraints, due to the prohibitive computational cost in processing long untrimmed videos. In this paper, we resolve this challenge by introducing a novel low-fidelity end-to-end (LoFi) video encoder pre-training method. Instead of always using the full training configurations for TAL learning, we propose to reduce the mini-batch composition in terms of temporal, spatial or spatio-temporal resolution so that end-to-end optimization for the video encoder becomes operable under the memory conditions of a mid-range hardware budget. Crucially, this enables the gradient to flow backward through the video encoder from a TAL loss supervision, favourably solving the task discrepancy problem and providing more effective feature representations. Extensive experiments show that the proposed LoFi pre-training approach can significantly enhance the performance of existing TAL methods. Encouragingly, even with a lightweight ResNet18 based video encoder in a single RGB stream, our method surpasses two-stream ResNet50 based alternatives with expensive optical flow, often by a good margin.

arXiv commentTo appear at NeurIPS 2021. 15 pages, 1 figure

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
• CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
• BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
• BMN: Boundary-Matching Network for Temporal Action Proposal Generation
• Rethinking the Hyperparameters for Fine-tuning
• TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
• CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
• CTAP: Complementary Temporal Action Proposal Generation
• Proposal Relation Network for Temporal Action Detection
• Dense Regression Network for Video Grounding
• Boundary-sensitive Pre-training for Temporal Localization in Videos