Multitask Text-to-Visual Embedding with Titles and Clickthrough Data
arXiv:1905.13339
Abstract
Text-visual (or called semantic-visual) embedding is a central problem in vision-language research. It typically involves mapping of an image and a text description to a common feature space through a CNN image encoder and a RNN language encoder. In this paper, we propose a new method for learning text-visual embedding using both image titles and click-through data from an image search engine. We also propose a new triplet loss function by modeling positive awareness of the embedding, and introduce a novel mini-batch-based hard negative sampling approach for better data efficiency in the learning process. Experimental results show that our proposed method outperforms existing methods, and is also effective for real-world text-to-visual retrieval.
4 pages. Language and Vision Workshop, in conjunction with CVPR 2019
References in corpus (5)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
- Learning Fine-grained Image Similarity with Deep Ranking
- CCL: Cross-modal Correlation Learning with Multi-grained Fusion by Hierarchical Network
- Revisiting Cross Modal Retrieval