Attentive Sequence to Sequence Translation for Localizing Clips of Interest by Natural Language Descriptions

Abstract

We propose a novel attentive sequence to sequence translator (ASST) for clip localization in videos by natural language descriptions. We make two contributions. First, we propose a bi-directional Recurrent Neural Network (RNN) with a finely calibrated vision-language attentive mechanism to comprehensively understand the free-formed natural language descriptions. The RNN parses natural language descriptions in two directions, and the attentive model attends every meaningful word or phrase to each frame, thereby resulting in a more detailed understanding of video content and description semantics. Second, we design a hierarchical architecture for the network to jointly model language descriptions and video content. Given a video-description pair, the network generates a matrix representation, i.e., a sequence of vectors. Each vector in the matrix represents a video frame conditioned by the description. The 2D representation not only preserves the temporal dependencies of frames but also provides an effective way to perform frame-level video-language matching. The hierarchical architecture exploits video content with multiple granularities, ranging from subtle details to global context. Integration of the multiple granularities yields a robust representation for multi-level video-language abstraction. We validate the effectiveness of our ASST on two large-scale datasets. Our ASST outperforms the state-of-the-art by $4.28\%$ in Rank$@1$ on the DiDeMo dataset. On the Charades-STA dataset, we significantly improve the state-of-the-art by $13.41\%$ in Rank$@1,IoU=0.5$.

References in corpus (28)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• WaveNet: A Generative Model for Raw Audio
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Convolutional Sequence to Sequence Learning
• Multi-Scale Context Aggregation by Dilated Convolutions
• Focal Loss for Dense Object Detection
• Language Modeling with Gated Convolutional Networks
• Feature Pyramid Networks for Object Detection
• Towards Good Practices for Very Deep Two-Stream ConvNets
• Beyond Skip Connections: Top-Down Modulation for Object Detection
• Neural Machine Translation in Linear Time
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Sequence to Sequence -- Video to Text
• Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs
• Speed/accuracy trade-offs for modern convolutional object detectors
• Learning Spatiotemporal Features with 3D Convolutional Networks
• R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
• Generation and Comprehension of Unambiguous Object Descriptions
• Temporal Context Network for Activity Localization in Videos
• TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
• CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
• Cascaded Boundary Regression for Temporal Action Detection
• Natural Language Object Retrieval
• Temporal Convolutional Networks for Action Segmentation and Detection