On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing

Abstract

Different languages might have different word orders. In this paper, we investigate cross-lingual transfer and posit that an order-agnostic model will perform better when transferring to distant foreign languages. To test our hypothesis, we train dependency parsers on an English corpus and evaluate their transfer performance on 30 other languages. Specifically, we compare encoders and decoders based on Recurrent Neural Networks (RNNs) and modified self-attentive architectures. The former relies on sequential information while the latter is more flexible at modeling word order. Rigorous experiments and detailed analysis shows that RNN-based architectures transfer well to languages that are close to English, while self-attentive models have better overall cross-lingual transferability and perform especially well on distant languages.

arXiv commentAccepted by NAACL-2019

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Multi-Task Cross-Lingual Sequence Tagging from Scratch
• Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields
• Neural Cross-Lingual Entity Linking

Cited by in corpus (2)

• Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers
• Density Matching for Bilingual Word Embedding