Predicting Visual Features from Text for Image and Video Caption Retrieval

Abstract

This paper strives to find amidst a set of sentences the one best describing the content of a given image or video. Different from existing works, which rely on a joint subspace for their image and video caption retrieval, we propose to do so in a visual space exclusively. Apart from this conceptual novelty, we contribute \emph{Word2VisualVec}, a deep neural network architecture that learns to predict a visual feature representation from textual input. Example captions are encoded into a textual embedding based on multi-scale sentence vectorization and further transferred into a deep visual feature of choice via a simple multi-layer perceptron. We further generalize Word2VisualVec for video caption retrieval, by predicting from text both 3-D convolutional neural network features as well as a visual-audio representation. Experiments on Flickr8k, Flickr30k, the Microsoft Video Description dataset and the very recent NIST TrecVid challenge for video caption retrieval detail Word2VisualVec's properties, its benefit over textual embeddings, the potential for multimodal query composition and its state-of-the-art results.

arXiv commentAccepted by Transaction on Multimedia. Code is available at https://github.com/danieljf24/w2vv

References in corpus (2)

• Describing Multimedia Content using Attention-based Encoder--Decoder Networks
• The ImageNet Shuffle: Reorganized Pre-training for Video Event Detection

Cited by in corpus (15)

• Dual Encoding for Video Retrieval by Text
• MDMMT: Multidomain Multimodal Transformer for Video Retrieval
• Kernelized Multiview Subspace Analysis by Self-weighted Learning
• Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
• Audio Retrieval with Natural Language Queries: A Benchmark Study
• Frame-wise Cross-modal Matching for Video Moment Retrieval
• Video Storytelling: Textual Summaries for Events
• Text-based Localization of Moments in a Video Corpus
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• Feature Re-Learning with Data Augmentation for Video Relevance Prediction
• Learning semantic sentence representations from visually grounded language without lexical knowledge
• Hybrid Contrastive Quantization for Efficient Cross-View Video Retrieval
• Text-Driven Video Acceleration: A Weakly-Supervised Reinforcement Learning Method
• PRVR: Partially Relevant Video Retrieval
• Migration-Related Semantic Concepts for the Retrieval of Relevant Video Content