Soft Prompt Decoding for Multilingual Dense Retrieval

Abstract

In this work, we explore a Multilingual Information Retrieval (MLIR) task, where the collection includes documents in multiple languages. We demonstrate that applying state-of-the-art approaches developed for cross-lingual information retrieval to MLIR tasks leads to sub-optimal performance. This is due to the heterogeneous and imbalanced nature of multilingual collections -- some languages are better represented in the collection and some benefit from large-scale training data. To address this issue, we present KD-SPD, a novel soft prompt decoding approach for MLIR that implicitly "translates" the representation of documents in different languages into the same embedding space. To address the challenges of data scarcity and imbalance, we introduce a knowledge distillation strategy. The teacher model is trained on rich English retrieval data, and by leveraging bi-text data, our distillation framework transfers its retrieval knowledge to the multilingual document encoder. Therefore, our approach does not require any multilingual retrieval training data. Extensive experiments on three MLIR datasets with a total of 15 languages demonstrate that KD-SPD significantly outperforms competitive baselines in all cases. We conduct extensive analyses to show that our method has less language bias and better zero-shot transfer ability towards new languages.

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• Knowledge Distillation: A Survey
• An Overview of Multi-Task Learning in Deep Neural Networks
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Universal Transformers
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
• Overview of the TREC 2020 deep learning track
• Deep Graph-level Anomaly Detection by Glocal Knowledge Distillation
• Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation
• On Transferability of Prompt Tuning for Natural Language Processing
• Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
• A Study of Neural Matching Models for Cross-lingual IR
• C3: Continued Pretraining with Contrastive Weak Supervision for Cross Language Ad-Hoc Retrieval
• Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation
• Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
• Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval
• Teaching a New Dog Old Tricks: Resurrecting Multilingual Retrieval Using Zero-shot Learning
• Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval
• Bitext Mining for Low-Resource Languages via Contrastive Learning
• Parameter-efficient Zero-shot Transfer for Cross-Language Dense Retrieval with Adapters