BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation

Abstract

Generating human action proposals in untrimmed videos is an important yet challenging task with wide applications. Current methods often suffer from the noisy boundary locations and the inferior quality of confidence scores used for proposal retrieving. In this paper, we present BSN++, a new framework which exploits complementary boundary regressor and relation modeling for temporal proposal generation. First, we propose a novel boundary regressor based on the complementary characteristics of both starting and ending boundary classifiers. Specifically, we utilize the U-shaped architecture with nested skip connections to capture rich contexts and introduce bi-directional boundary matching mechanism to improve boundary precision. Second, to account for the proposal-proposal relations ignored in previous methods, we devise a proposal relation block to which includes two self-attention modules from the aspects of position and channel. Furthermore, we find that there inevitably exists data imbalanced problems in the positive/negative proposals and temporal durations, which harm the model performance on tail distributions. To relieve this issue, we introduce the scale-balanced re-sampling strategy. Extensive experiments are conducted on two popular benchmarks: ActivityNet-1.3 and THUMOS14, which demonstrate that BSN++ achieves the state-of-the-art performance. Not surprisingly, the proposed BSN++ ranked 1st place in the CVPR19 - ActivityNet challenge leaderboard on temporal action localization task.

arXiv commentAccepted by AAAI 2021. Ranked 1st place in the CVPR19 - ActivityNet Challenge leaderboard on Temporal Action Localization task. arXiv admin note: substantial text overlap with arXiv:2007.09883

References in corpus (14)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• UNet++: A Nested U-Net Architecture for Medical Image Segmentation
• A Pursuit of Temporal Accuracy in General Activity Detection
• CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
• Untrimmed Video Classification for Activity Detection: submission to ActivityNet Challenge
• BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
• BMN: Boundary-Matching Network for Temporal Action Proposal Generation
• Temporal Convolution Based Action Proposal: Submission to ActivityNet 2017
• CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
• Cascaded Boundary Regression for Temporal Action Detection
• CTAP: Complementary Temporal Action Proposal Generation
• Class-wise Dynamic Graph Convolution for Semantic Segmentation
• Collaborative Distillation in the Parameter and Spectrum Domains for Video Action Recognition

Cited by in corpus (7)

• RGB Stream Is Enough for Temporal Action Detection
• Temporal Action Proposal Generation with Transformers
• ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation
• Temporal Context Aggregation Network for Temporal Action Proposal Refinement
• AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation
• Joint Multimedia Event Extraction from Video and Article
• TSI: Temporal Saliency Integration for Video Action Recognition