Temporal Dynamic Graph LSTM for Action-driven Video Object Detection

Abstract

In this paper, we investigate a weakly-supervised object detection framework. Most existing frameworks focus on using static images to learn object detectors. However, these detectors often fail to generalize to videos because of the existing domain shift. Therefore, we investigate learning these detectors directly from boring videos of daily activities. Instead of using bounding boxes, we explore the use of action descriptions as supervision since they are relatively easy to gather. A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label". To tackle this problem, we propose a novel temporal dynamic graph Long Short-Term Memory network (TD-Graph LSTM). TD-Graph LSTM enables global temporal reasoning by constructing a dynamic graph that is based on temporal correlations of object proposals and spans the entire video. The missing label issue for each individual frame can thus be significantly alleviated by transferring knowledge across correlated objects proposals in the whole video. Extensive evaluations on a large-scale daily-life action dataset (i.e., Charades) demonstrates the superiority of our proposed method. We also release object bounding-box annotations for more than 5,000 frames in Charades. We believe this annotated data can also benefit other research on video-based object recognition in the future.

To appear in ICCV 2017

References in corpus (13)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Weakly Supervised Object Localization with Multi-fold Multiple Instance Learning
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Unsupervised Learning of Visual Representations using Videos
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Video Summarization with Long Short-term Memory
• ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
• End-to-end Learning of Action Detection from Frame Glimpses in Videos
• Semantic Object Parsing with Local-Global Long Short-Term Memory
• Deep Self-Taught Learning for Weakly Supervised Object Localization
• Interpretable Structure-Evolving LSTM
• Unsupervised Object Discovery and Tracking in Video Collections

Cited by in corpus (9)

• Dynamic Graph Convolutional Networks
• Deep Learning in Mobile and Wireless Networking: A Survey
• Gated Fusion Network for Single Image Dehazing
• Learning Human-Object Interactions by Graph Parsing Neural Networks
• Revisiting Dilated Convolution: A Simple Approach for Weakly- and Semi- Supervised Semantic Segmentation
• Bidirectional Attentive Fusion with Context Gating for Dense Video Captioning
• PeerNets: Exploiting Peer Wisdom Against Adversarial Attacks
• Learning Graph-Level Representations with Recurrent Neural Networks
• Adaptive Temporal Encoding Network for Video Instance-level Human Parsing