A Strong and Robust Baseline for Text-Image Matching

Abstract

We review the current schemes of text-image matching models and propose improvements for both training and inference. First, we empirically show limitations of two popular loss (sum and max-margin loss) widely used in training text-image embeddings and propose a trade-off: a kNN-margin loss which 1) utilizes information from hard negatives and 2) is robust to noise as all $K$-most hardest samples are taken into account, tolerating \emph{pseudo} negatives and outliers. Second, we advocate the use of Inverted Softmax (\textsc{Is}) and Cross-modal Local Scaling (\textsc{Csls}) during inference to mitigate the so-called hubness problem in high-dimensional embedding space, enhancing scores of all metrics by a large margin.

6 pages (excluding references); 2019 ACL Student Research Workshop (to appear)

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Finding beans in burgers: Deep semantic-visual embedding with localization
• Phrase-based Image Captioning
• Learning Visually-Grounded Semantics from Contrastive Adversarial Samples