Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos
arXiv:1901.06829
Abstract
The task of video grounding, which temporally localizes a natural language description in a video, plays an important role in understanding videos. Existing studies have adopted strategies of sliding window over the entire video or exhaustively ranking all possible clip-sentence pairs in a pre-segmented video, which inevitably suffer from exhaustively enumerated candidates. To alleviate this problem, we formulate this task as a problem of sequential decision making by learning an agent which regulates the temporal grounding boundaries progressively based on its policy. Specifically, we propose a reinforcement learning based framework improved by multi-task learning and it shows steady performance gains by considering additional supervised boundary information during training. Our proposed framework achieves state-of-the-art performance on ActivityNet'18 DenseCaption dataset and Charades-STA dataset while observing only 10 or less clips per video.
arXiv commentAAAI 2019
References in corpus (10)
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- Temporal Action Detection with Structured Segment Networks
- TALL: Temporal Activity Localization via Language Query
- Recurrent Topic-Transition GAN for Visual Paragraph Generation
- CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- Weakly-Supervised Alignment of Video With Text
- Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
- ACD: Action Concept Discovery from Image-Sentence Corpora