mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
arXiv:2108.13897
Abstract
The MS MARCO ranking dataset has been widely used for training deep learning models for IR tasks, achieving considerable effectiveness on diverse zero-shot scenarios. However, this type of resource is scarce in languages other than English. In this work, we present mMARCO, a multilingual version of the MS MARCO passage ranking dataset comprising 13 languages that was created using machine translation. We evaluated mMARCO by finetuning monolingual and multilingual reranking models, as well as a multilingual dense retrieval model on this dataset. We also evaluated models finetuned using the mMARCO dataset in a zero-shot scenario on Mr. TyDi dataset, demonstrating that multilingual models finetuned on our translated dataset achieve superior effectiveness to models finetuned on the original English version alone. Our experiments also show that a distilled multilingual reranker is competitive with non-distilled models while having 5.4 times fewer parameters. Lastly, we show a positive correlation between translation quality and retrieval effectiveness, providing evidence that improvements in translation methods might lead to improvements in multilingual information retrieval. The translated datasets and finetuned models are available at https://github.com/unicamp-dl/mMARCO.
References in corpus (15)
- Exploiting Similarities among Languages for Machine Translation
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- On the Cross-lingual Transferability of Monolingual Representations
- Massively Multilingual Word Embeddings
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Multi-Stage Document Ranking with BERT
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
- Overview of the TREC 2019 deep learning track
- RepBERT: Contextualized Text Embeddings for First-Stage Retrieval
- Overview of the TREC 2021 deep learning track
- The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
- Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering
- TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
- PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
- Should we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?