A Comprehensive Survey on Cross-modal Retrieval

Abstract

In recent years, cross-modal retrieval has drawn much attention due to the rapid growth of multimodal data. It takes one type of data as the query to retrieve relevant data of another type. For example, a user can use a text to retrieve relevant pictures or videos. Since the query and its retrieved results can be of different modalities, how to measure the content similarity between different modalities of data remains a challenge. Various methods have been proposed to deal with such a problem. In this paper, we first review a number of representative methods for cross-modal retrieval and classify them into two main groups: 1) real-valued representation learning, and 2) binary representation learning. Real-valued representation learning methods aim to learn real-valued common representations for different modalities of data. To speed up the cross-modal retrieval, a number of binary representation learning methods are proposed to map different modalities of data into a common Hamming space. Then, we introduce several multimodal datasets in the community, and show the experimental results on two commonly used multimodal datasets. The comparison reveals the characteristic of different kinds of cross-modal retrieval methods, which is expected to benefit both practical applications and future research. Finally, we discuss open problems and future research directions.

20 pages, 11 figures, 9 tables

References in corpus (2)

• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• A Survey on Multi-view Learning

Cited by in corpus (33)

• Deep Learning Techniques for Future Intelligent Cross-Media Retrieval
• From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
• Representation Learning for Natural Language Processing
• BadCM: Invisible Backdoor Attack Against Cross-Modal Learning
• Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval
• Self-supervised learning of visual features through embedding images into text topic spaces
• HashGAN:Attention-aware Deep Adversarial Hashing for Cross Modal Retrieval
• Multimodal Neural Databases
• Beyond Intra-modality: A Survey of Heterogeneous Person Re-identification
• Cross-modal Variational Auto-encoder for Content-based Micro-video Background Music Recommendation
• Learning Multi-Modal Nonlinear Embeddings: Performance Bounds and an Algorithm
• Quantum-inspired canonical correlation analysis for exponentially large dimensional data
• Cross-Modal Retrieval with Implicit Concept Association
• Deep Cross-Modal Audio-Visual Generation
• When the Music Stops: Tip-of-the-Tongue Retrieval for Music
• Temporal Cross-Media Retrieval with Soft-Smoothing
• Emotion Embedding Spaces for Matching Music to Stories
• TextTopicNet - Self-Supervised Learning of Visual Features Through Embedding Images on Semantic Text Spaces
• Cross-modal Variational Auto-encoder with Distributed Latent Spaces and Associators
• Self-Supervised Visual Representations for Cross-Modal Retrieval
• HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs
• Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval
• Robust Matrix Elastic Net based Canonical Correlation Analysis: An Effective Algorithm for Multi-View Unsupervised Learning
• Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization
• Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretraining
• Comprehensive Graph-conditional Similarity Preserving Network for Unsupervised Cross-modal Hashing
• From Multimodal to Unimodal Webpages for Developing Countries
• Modal-aware Features for Multimodal Hashing
• SwAMP: Swapped Assignment of Multi-Modal Pairs for Cross-Modal Retrieval
• Error-Corrected Margin-Based Deep Cross-Modal Hashing for Facial Image Retrieval
• Multimodal Information Retrieval for Open World with Edit Distance Weak Supervision
• Graph Pattern Loss based Diversified Attention Network for Cross-Modal Retrieval
• From Text to Sound: A Preliminary Study on Retrieving Sound Effects to Radio Stories