An Efficient Approach for Geo-Multimedia Cross-Modal Retrieval

Abstract

Due to the rapid development of mobile Internet techniques, cloud computation and popularity of online social networking and location-based services, massive amount of multimedia data with geographical information is generated and uploaded to the Internet. In this paper, we propose a novel type of cross-modal multimedia retrieval called geo-multimedia cross-modal retrieval which aims to search out a set of geo-multimedia objects based on geographical distance proximity and semantic similarity between different modalities. Previous studies for cross-modal retrieval and spatial keyword search cannot address this problem effectively because they do not consider multimedia data with geo-tags and do not focus on this type of query. In order to address this problem efficiently, we present the definition of $k$NN geo-multimedia cross-modal query at the first time and introduce relevant conceptions such as cross-modal semantic representation space. To bridge the semantic gap between different modalities, we propose a method named cross-modal semantic matching which contains two important component, i.e., CorrProj and LogsTran, which aims to construct a common semantic representation space for cross-modal semantic similarity measurement. Besides, we designed a framework based on deep learning techniques to implement common semantic representation space construction. In addition, a novel hybrid indexing structure named GMR-Tree combining geo-multimedia data and R-Tree is presented and a efficient $k$NN search algorithm called $k$GMCMS is designed. Comprehensive experimental evaluation on real and synthetic dataset clearly demonstrates that our solution outperforms the-state-of-the-art methods.

arXiv comment27 pages

References in corpus (14)

• Reasoning about Entailment with Neural Attention
• Multi-View Spectral Clustering via Structured Low-Rank Matrix Factorization
• Deep Adaptive Feature Embedding with Local Sample Distributions for Person Re-identification
• Learning to Compare Image Patches via Convolutional Neural Networks
• Effective Multi-Query Expansions: Collaborative Deep Networks for Robust Landmark Retrieval
• Sentence Similarity Learning by Lexical Decomposition and Composition
• A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
• What-and-Where to Match: Deep Spatially Multiplicative Integration Networks for Person Re-identification
• Multimodal Machine Learning: A Survey and Taxonomy
• Iterative Views Agreement: An Iterative Low-Rank based Structured Optimization Method to Multi-View Spectral Clustering
• Structured Deep Hashing with Convolutional Neural Networks for Fast Person Re-identification
• Learning Semantic Textual Similarity from Conversations
• Learning Social Image Embedding with Deep Multimodal Attention Networks
• Developing a comprehensive framework for multimodal feature extraction