Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond

Abstract

Cross-lingual adaptation with multilingual pre-trained language models (mPTLMs) mainly consists of two lines of works: zero-shot approach and translation-based approach, which have been studied extensively on the sequence-level tasks. We further verify the efficacy of these cross-lingual adaptation approaches by evaluating their performances on more fine-grained sequence tagging tasks. After re-examining their strengths and drawbacks, we propose a novel framework to consolidate the zero-shot approach and the translation-based approach for better adaptation performance. Instead of simply augmenting the source data with the machine-translated data, we tailor-make a warm-up mechanism to quickly update the mPTLMs with the gradients estimated on a few translated data. Then, the adaptation approach is applied to the refined parameters and the cross-lingual transfer is performed in a warm-start way. The experimental results on nine target languages demonstrate that our method is beneficial to the cross-lingual adaptation of various sequence tagging tasks.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• An Overview of Multi-Task Learning in Deep Neural Networks
• Unsupervised Data Augmentation for Consistency Training
• Exploiting Similarities among Languages for Machine Translation
• On the Cross-lingual Transferability of Monolingual Representations
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Offline bilingual word vectors, orthogonal transformations and the inverted softmax
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Multilingual Alignment of Contextual Word Representations
• Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpus
• Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources

Cited by in corpus (3)

• Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challenges
• QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction
• Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding