Jointly Modeling Embedding and Translation to Bridge Video and Language

Abstract

Automatically describing video content with natural language is a fundamental challenge of multimedia. Recurrent Neural Networks (RNN), which models sequence dynamics, has attracted increasing attention on visual interpretation. However, most existing approaches generate a word locally with given previous words and the visual content, while the relationship between sentence semantics and visual content is not holistically exploited. As a result, the generated sentences may be contextually correct but the semantics (e.g., subjects, verbs or objects) are not true. This paper presents a novel unified framework, named Long Short-Term Memory with visual-semantic Embedding (LSTM-E), which can simultaneously explore the learning of LSTM and visual-semantic embedding. The former aims to locally maximize the probability of generating the next word given previous words and visual content, while the latter is to create a visual-semantic embedding space for enforcing the relationship between the semantics of the entire sentence and visual content. Our proposed LSTM-E consists of three components: a 2-D and/or 3-D deep convolutional neural networks for learning powerful video representation, a deep RNN for generating sentences, and a joint embedding model for exploring the relationships between visual content and sentence semantics. The experiments on YouTube2Text dataset show that our proposed LSTM-E achieves to-date the best reported performance in generating natural sentences: 45.3% and 31.0% in terms of BLEU@4 and METEOR, respectively. We also demonstrate that LSTM-E is superior in predicting Subject-Verb-Object (SVO) triplets to several state-of-the-art techniques.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Going Deeper with Convolutions
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Explain Images with Multimodal Recurrent Neural Networks
• Learning to Execute
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• Learning Spatiotemporal Features with 3D Convolutional Networks

Cited by in corpus (22)

• A Closer Look at Spatiotemporal Convolutions for Action Recognition
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• TGIF: A New Dataset and Benchmark on Animated GIF Description
• Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction
• Video-based Sign Language Recognition without Temporal Segmentation
• Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
• Weakly Supervised Dense Video Captioning
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• Spatio-Temporal Attention Models for Grounded Video Captioning
• Attention-Based Multimodal Fusion for Video Description
• Video Captioning via Hierarchical Reinforcement Learning
• Semantic Compositional Networks for Visual Captioning
• Improving Interpretability of Deep Neural Networks with Semantic Information
• Bidirectional Long-Short Term Memory for Video Description
• Attend and Interact: Higher-Order Object Interactions for Video Understanding
• Multimodal Memory Modelling for Video Captioning
• Learning long-term dependencies for action recognition with a biologically-inspired deep network
• Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text Recognition
• COBE: Contextualized Object Embeddings from Narrated Instructional Video
• Middle-Out Decoding
• Bidirectional Multirate Reconstruction for Temporal Modeling in Videos
• The Long-Short Story of Movie Description