Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
arXiv:1812.10464 · doi:10.1162/tacl_a_00288
Abstract
We introduce an architecture to learn joint multilingual sentence representations for 93 languages, belonging to more than 30 different families and written in 28 different scripts. Our system uses a single BiLSTM encoder with a shared BPE vocabulary for all languages, which is coupled with an auxiliary decoder and trained on publicly available parallel corpora. This enables us to learn a classifier on top of the resulting embeddings using English annotated data only, and transfer it to any of the 93 languages without any modification. Our experiments in cross-lingual natural language inference (XNLI dataset), cross-lingual document classification (MLDoc dataset) and parallel corpus mining (BUCC dataset) show the effectiveness of our approach. We also introduce a new test set of aligned sentences in 112 languages, and show that our sentence embeddings obtain strong results in multilingual similarity search even for low-resource languages. Our implementation, the pre-trained encoder and the multilingual test set are available at https://github.com/facebookresearch/LASER
TACL
References in corpus (3)
Cited by in corpus (28)
- Cross-lingual Language Model Pretraining
- How multilingual is Multilingual BERT?
- Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
- GREEK-BERT: The Greeks visiting Sesame Street
- PMIndia -- A Collection of Parallel Corpora of Languages of India
- Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT
- Cross-lingual Zero- and Few-shot Hate Speech Detection Utilising Frozen Transformer Language Models and AXEL
- Massively Multilingual Neural Machine Translation
- A Comprehensive Survey of Multilingual Neural Machine Translation
- Bridging the domain gap in cross-lingual document classification
- Multilingual Question Answering from Formatted Text applied to Conversational Agents
- CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP
- Evaluation of Sentence Representations in Polish
- MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
- Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian
- Towards Zero-shot Cross-lingual Image Retrieval and Tagging
- Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER
- A Survey of Embedding Space Alignment Methods for Language and Knowledge Graphs
- Learning Bilingual Sentence Embeddings via Autoencoding and Computing Similarities with a Multilayer Perceptron
- Facebook AI's WAT19 Myanmar-English Translation Task Submission
- Simple and Effective Paraphrastic Similarity from Parallel Translations
- LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Models
- Aligning Cross-Lingual Entities with Multi-Aspect Information
- Bot-Match: Social Bot Detection with Recursive Nearest Neighbors Search
- ABSent: Cross-Lingual Sentence Representation Mapping with Bidirectional GANs
- Training Cross-Lingual embeddings for Setswana and Sepedi
- COSTRA 1.0: A Dataset of Complex Sentence Transformations
- Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification