Aligning Cross-lingual Sentence Representations with Dual Momentum Contrast
arXiv:2109.00253
Abstract
In this paper, we propose to align sentence representations from different languages into a unified embedding space, where semantic similarities (both cross-lingual and monolingual) can be computed with a simple dot product. Pre-trained language models are fine-tuned with the translation ranking task. Existing work (Feng et al., 2020) uses sentences within the same batch as negatives, which can suffer from the issue of easy negatives. We adapt MoCo (He et al., 2020) to further improve the quality of alignment. As the experimental results show, the sentence representations produced by our model achieve the new state-of-the-art on several tasks, including Tatoeba en-zh similarity search (Artetxe and Schwenk, 2019b), BUCC en-zh bitext mining, and semantic textual similarity on 7 datasets.
arXiv commentAccepted to EMNLP 2021 main conference
References in corpus (11)
- Learning Transferable Visual Models From Natural Language Supervision
- Bootstrap your own latent: A new approach to self-supervised Learning
- Improved Baselines with Momentum Contrastive Learning
- Contrastive Representation Learning: A Framework and Review
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
- Whitening Sentence Representations for Better Semantics and Faster Retrieval
- DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
- On the Sentence Embeddings from Pre-trained Language Models
- VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation