Dual-Path Convolutional Image-Text Embeddings with Instance Loss
arXiv:1711.05535 · doi:10.1145/3383184
Abstract
Matching images and sentences demands a fine understanding of both modalities. In this paper, we propose a new system to discriminatively embed the image and text to a shared visual-textual space. In this field, most existing works apply the ranking loss to pull the positive image / text pairs close and push the negative pairs apart from each other. However, directly deploying the ranking loss is hard for network learning, since it starts from the two heterogeneous features to build inter-modal relationship. To address this problem, we propose the instance loss which explicitly considers the intra-modal data distribution. It is based on an unsupervised assumption that each image / text group can be viewed as a class. So the network can learn the fine granularity from every image/text group. The experiment shows that the instance loss offers better weight initialization for the ranking loss, so that more discriminative embeddings can be learned. Besides, existing works usually apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So in a minor contribution, this paper constructs an end-to-end dual-path convolutional network to learn the image and text representations. End-to-end learning allows the system to directly learn from the data and fully utilize the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO), experiments demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language based person retrieval, we improve the state of the art by a large margin. The code has been made publicly available.
arXiv comment15pages, 15 figures, 8 tables
References in corpus (4)
Cited by in corpus (26)
- Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization
- Multi-Modal Knowledge Graph Construction and Application: A Survey
- A Transformer-Based Feature Segmentation and Region Alignment Method For UAV-View Geo-Localization
- VehicleNet: Learning Robust Visual Representation for Vehicle Re-identification
- CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval
- Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifold
- A New Benchmark and Approach for Fine-grained Cross-media Retrieval
- Parameter-Efficient Person Re-identification in the 3D Space
- Offline-Online Associated Camera-Aware Proxies for Unsupervised Person Re-identification
- Geo-Localization via Ground-to-Satellite Cross-View Image Retrieval
- Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
- Bridging the Gap: Multi-Level Cross-Modality Joint Alignment for Visible-Infrared Person Re-Identification
- Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval
- ALR-GAN: Adaptive Layout Refinement for Text-to-Image Synthesis
- Style Alignment based Dynamic Observation Method for UAV-View Geo-localization
- Learnable Pillar-based Re-ranking for Image-Text Retrieval
- From Data Deluge to Data Curation: A Filtering-WoRA Paradigm for Efficient Text-based Person Search
- Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval
- Hierarchical Matching and Reasoning for Multi-Query Image Retrieval
- Multi-modal Reference Learning for Fine-grained Text-to-Image Retrieval
- Prototype-guided Cross-modal Completion and Alignment for Incomplete Text-based Person Re-identification
- Taking Modality-free Human Identification as Zero-shot Learning
- Collaborative Group: Composed Image Retrieval via Consensus Learning from Noisy Annotations
- Real-Time Non-Invasive Imaging and Detection of Spreading Depolarizations through EEG: An Ultra-Light Explainable Deep Learning Approach
- SCMM: Calibrating Cross-modal Representations for Text-Based Person Search
- DAPL: Integration of Positive and Negative Descriptions in Text-Based Person Search