OadTR: Online Action Detection with Transformers

Abstract

Most recent approaches for online action detection tend to apply Recurrent Neural Network (RNN) to capture long-range temporal structure. However, RNN suffers from non-parallelism and gradient vanishing, hence it is hard to be optimized. In this paper, we propose a new encoder-decoder framework based on Transformers, named OadTR, to tackle these problems. The encoder attached with a task token aims to capture the relationships and global interactions between historical observations. The decoder extracts auxiliary information by aggregating anticipated future clip representations. Therefore, OadTR can recognize current actions by encoding historical information and predicting future context simultaneously. We extensively evaluate the proposed OadTR on three challenging datasets: HDD, TVSeries, and THUMOS14. The experimental results show that OadTR achieves higher training and inference speeds than current RNN based approaches, and significantly outperforms the state-of-the-art methods in terms of both mAP and mcAP. Code is available at https://github.com/wangxiang1230/OadTR.

arXiv commentCode is available at https://github.com/wangxiang1230/OadTR

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Big Bird: Transformers for Longer Sequences
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
• R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
• End-to-End Object Detection with Adaptive Clustering Transformer
• ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis
• Neural Speech Synthesis with Transformer Network
• RED: Reinforced Encoder-Decoder Networks for Action Anticipation
• Self-trained Deep Ordinal Regression for End-to-End Video Anomaly Detection
• Temporal Recurrent Networks for Online Action Detection
• Boundary Content Graph Neural Network for Temporal Action Proposal Generation
• Learning to Discriminate Information for Online Action Detection