Soft Prompt Decoding for Multilingual Dense Retrieval

Abstract

In this work, we explore a Multilingual Information Retrieval (MLIR) task, where the collection includes documents in multiple languages. We demonstrate that applying state-of-the-art approaches developed for cross-lingual information retrieval to MLIR tasks leads to sub-optimal performance. This is due to the heterogeneous and imbalanced nature of multilingual collections -- some languages are better represented in the collection and some benefit from large-scale training data. To address this issue, we present KD-SPD, a novel soft prompt decoding approach for MLIR that implicitly "translates" the representation of documents in different languages into the same embedding space. To address the challenges of data scarcity and imbalance, we introduce a knowledge distillation strategy. The teacher model is trained on rich English retrieval data, and by leveraging bi-text data, our distillation framework transfers its retrieval knowledge to the multilingual document encoder. Therefore, our approach does not require any multilingual retrieval training data. Extensive experiments on three MLIR datasets with a total of 15 languages demonstrate that KD-SPD significantly outperforms competitive baselines in all cases. We conduct extensive analyses to show that our method has less language bias and better zero-shot transfer ability towards new languages.

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• An Overview of Multi-Task Learning in Deep Neural Networks
• Overview of the TREC 2020 deep learning track
• Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation
• Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
• A Study of Neural Matching Models for Cross-lingual IR
• C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval
• Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
• Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation
• Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval
• Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
• Bitext Mining for Low-Resource Languages via Contrastive Learning