Temporal Recurrent Networks for Online Action Detection

Abstract

Most work on temporal action detection is formulated as an offline problem, in which the start and end times of actions are determined after the entire video is fully observed. However, important real-time applications including surveillance and driver assistance systems require identifying actions as soon as each video frame arrives, based only on current and historical observations. In this paper, we propose a novel framework, Temporal Recurrent Network (TRN), to model greater temporal context of a video frame by simultaneously performing online action detection and anticipation of the immediate future. At each moment in time, our approach makes use of both accumulated historical evidence and predicted future information to better recognize the action that is currently occurring, and integrates both of these into a unified end-to-end architecture. We evaluate our approach on two popular online action detection datasets, HDD and TVSeries, as well as another widely used dataset, THUMOS'14. The results show that TRN significantly outperforms the state-of-the-art.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016

Cited by in corpus (5)

• LAP-Net: Adaptive Features Sampling via Learning Action Progression for Online Action Detection
• StartNet: Online Detection of Action Start in Untrimmed Videos
• WSLLN: Weakly Supervised Natural Language Localization Networks
• OadTR: Online Action Detection with Transformers
• Goal-oriented Object Importance Estimation in On-road Driving Videos