Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning

Abstract

Despite the recent developments in the field of cross-modal retrieval, there has been less research focusing on low-resource languages due to the lack of manually annotated datasets. In this paper, we propose a noise-robust cross-lingual cross-modal retrieval method for low-resource languages. To this end, we use Machine Translation (MT) to construct pseudo-parallel sentence pairs for low-resource languages. However, as MT is not perfect, it tends to introduce noise during translation, rendering textual embeddings corrupted and thereby compromising the retrieval performance. To alleviate this, we introduce a multi-view self-distillation method to learn noise-robust target-language representations, which employs a cross-attention module to generate soft pseudo-targets to provide direct supervision from the similarity-based view and feature-based view. Besides, inspired by the back-translation in unsupervised MT, we minimize the semantic discrepancies between origin sentences and back-translated sentences to further improve the noise robustness of the textual encoder. Extensive experiments are conducted on three video-text and image-text cross-modal retrieval benchmarks across different languages, and the results demonstrate that our method significantly improves the overall performance without using extra human-labeled data. In addition, equipped with a pre-trained visual encoder from a recent vision-and-language pre-training framework, i.e., CLIP, our model achieves a significant performance gain, showing that our method is compatible with popular pre-training models. Code and data are available at https://github.com/HuiGuanLab/nrccr.

arXiv commentAccepted by ACM MM 2022. Code and data are available at https://github.com/HuiGuanLab/nrccr

References in corpus (19)

• Learning Transferable Visual Models From Natural Language Supervision
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Use What You Have: Video Retrieval Using Representations From Collaborative Experts
• Dual Encoding for Video Retrieval by Text
• Predicting Visual Features from Text for Image and Video Caption Retrieval
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval
• Imagination improves Multimodal Translation
• Fluency-Guided Cross-Lingual Image Captioning
• Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
• VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation
• Towards Zero-shot Cross-lingual Image Retrieval
• Image search using multilingual texts: a cross-modal learning approach between image and text
• HANet: Hierarchical Alignment Networks for Video-Text Retrieval
• CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising

Cited by in corpus (4)

• MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning
• From Region to Patch: Attribute-Aware Foreground-Background Contrastive Learning for Fine-Grained Fashion Retrieval
• Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learning
• An Empirical Study of Excitation and Aggregation Design Adaptions in CLIP4Clip for Video-Text Retrieval