Long Short-Term Relation Networks for Video Action Detection

Abstract

It has been well recognized that modeling human-object or object-object relations would be helpful for detection task. Nevertheless, the problem is not trivial especially when exploring the interactions between human actor, object and scene (collectively as human-context) to boost video action detectors. The difficulty originates from the aspect that reliable relations in a video should depend on not only short-term human-context relation in the present clip but also the temporal dynamics distilled over a long-range span of the video. This motivates us to capture both short-term and long-term relations in a video. In this paper, we present a new Long Short-Term Relation Networks, dubbed as LSTR, that novelly aggregates and propagates relation to augment features for video action detection. Technically, Region Proposal Networks (RPN) is remoulded to first produce 3D bounding boxes, i.e., tubelets, in each video clip. LSTR then models short-term human-context interactions within each clip through spatio-temporal attention mechanism and reasons long-term temporal dynamics across video clips via Graph Convolutional Networks (GCN) in a cascaded manner. Extensive experiments are conducted on four benchmark datasets, and superior results are reported when comparing to state-of-the-art methods.

Accepted as a full paper for ACMMM 2019

References in corpus (14)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• Seq-NMS for Video Object Detection
• Flow-Guided Feature Aggregation for Video Object Detection
• Learning to track for spatio-temporal action localization
• Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos
• Learning Human-Object Interactions by Graph Parsing Neural Networks
• Relation Distillation Networks for Video Object Detection
• Spatio-Temporal Action Detection with Cascade Proposal and Location Anticipation
• Actor-Centric Relation Network