Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
arXiv:2103.08849
Abstract
This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (MultiHowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is available at http://github.com/berniebear/Multi-HT100M.
arXiv commentaccepted by NAACL 2021
References in corpus (8)
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Billion-scale semi-supervised learning for image classification
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
- Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
- Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
- Video Understanding as Machine Translation