Temporal Action Proposal Generation with Transformers

Abstract

Transformer networks are effective at modeling long-range contextual information and have recently demonstrated exemplary performance in the natural language processing domain. Conventionally, the temporal action proposal generation (TAPG) task is divided into two main sub-tasks: boundary prediction and proposal confidence prediction, which rely on the frame-level dependencies and proposal-level relationships separately. To capture the dependencies at different levels of granularity, this paper intuitively presents a unified temporal action proposal generation framework with original Transformers, called TAPG Transformer, which consists of a Boundary Transformer and a Proposal Transformer. Specifically, the Boundary Transformer captures long-term temporal dependencies to predict precise boundary information and the Proposal Transformer learns the rich inter-proposal relationships for reliable confidence evaluation. Extensive experiments are conducted on two popular benchmarks: ActivityNet-1.3 and THUMOS14, and the results demonstrate that TAPG Transformer outperforms state-of-the-art methods. Equipped with the existing action classifier, our method achieves remarkable performance on the temporal action localization task. Codes and models will be available.

arXiv commentThe first three authors contributed equally

References in corpus (10)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Is Space-Time Attention All You Need for Video Understanding?
• Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
• Temporal Action Detection with Structured Segment Networks
• R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
• Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge
• TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
• CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
• TFPose: Direct Human Pose Estimation with Transformers
• BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation

Cited by in corpus (4)

• End-to-end Temporal Action Detection with Transformer
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
• Towards High-Quality Temporal Action Detection with Sparse Proposals
• Good Practices and A Strong Baseline for Traffic Anomaly Detection