Cross-Modal Retrieval with Implicit Concept Association

Abstract

Traditional cross-modal retrieval assumes explicit association of concepts across modalities, where there is no ambiguity in how the concepts are linked to each other, e.g., when we do the image search with a query "dogs", we expect to see dog images. In this paper, we consider a different setting for cross-modal retrieval where data from different modalities are implicitly linked via concepts that must be inferred by high-level reasoning; we call this setting implicit concept association. To foster future research in this setting, we present a new dataset containing 47K pairs of animated GIFs and sentences crawled from the web, in which the GIFs depict physical or emotional reactions to the scenarios described in the text (called "reaction GIFs"). We report on a user study showing that, despite the presence of implicit concept association, humans are able to identify video-sentence pairs with matching concepts, suggesting the feasibility of our task. Furthermore, we propose a novel visual-semantic embedding network based on multiple instance learning. Unlike traditional approaches, we compute multiple embeddings from each modality, each representing different concepts, and measure their similarity by considering all possible combinations of visual-semantic embeddings in the framework of multiple instance learning. We evaluate our approach on two video-sentence datasets with explicit and implicit concept association and report competitive results compared to existing approaches on cross-modal retrieval.

References in corpus (10)

• Recurrent Neural Network Regularization
• A Structured Self-attentive Sentence Embedding
• Improved Techniques for Training GANs
• Recurrent Models of Visual Attention
• A Comprehensive Survey on Cross-modal Retrieval
• TGIF: A New Dataset and Benchmark on Animated GIF Description
• Video2GIF: Automatic Generation of Animated GIFs from Video
• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• Learning Robust Visual-Semantic Embeddings
• Image2GIF: Generating Cinemagraphs using Recurrent Deep Q-Networks

Cited by in corpus (4)

• Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
• Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences
• ShapeCaptioner: Generative Caption Network for 3D Shapes by Learning a Mapping from Parts Detected in Multiple Views to Sentences
• Diff-3DCap: Shape Captioning with Diffusion Models