Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Abstract

This paper investigates the problem of learning cross-lingual representations in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextualized word embeddings based on publicly available pre-trained BERT models (Devlin et al., 2018). In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages. We demonstrate the effectiveness of this approach on zero-shot cross-lingual transfer parsing. Experiments show that our embeddings substantially outperform the previous state-of-the-art that uses static embeddings. We further compare our approach with XLM (Lample and Conneau, 2019), a recently proposed cross-lingual language model trained with massive parallel data, and achieve highly competitive results.

arXiv commentto appear at EMNLP 2019

References in corpus (5)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Exploiting Similarities among Languages for Machine Translation
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• How multilingual is Multilingual BERT?
• Learning Bilingual Word Representations by Marginalizing Alignments

Cited by in corpus (8)

• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Self-Attentive Constituency Parsing for UCCA-based Semantic Parsing
• Toward Cross-Lingual Definition Generation for Language Learners
• Emergent Properties of Finetuned Language Representation Models
• Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings