Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment

Abstract

The cross-lingual language models are typically pretrained with masked language modeling on multilingual text or parallel sentences. In this paper, we introduce denoising word alignment as a new cross-lingual pre-training task. Specifically, the model first self-labels word alignments for parallel sentences. Then we randomly mask tokens in a bitext pair. Given a masked token, the model uses a pointer network to predict the aligned token in the other language. We alternately perform the above two steps in an expectation-maximization manner. Experimental results show that our method improves cross-lingual transferability on various datasets, especially on the token-level tasks, such as question answering, and structured prediction. Moreover, the model can serve as a pretrained word aligner, which achieves reasonably low error rates on the alignment benchmarks. The code and pretrained parameters are available at https://github.com/CZWin32768/XLM-Align.

arXiv commentACL 2021

References in corpus (10)

• mT5: A massively multilingual pre-trained text-to-text transformer
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Multilingual Alignment of Contextual Word Representations
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• Inducing Language-Agnostic Multilingual Representations
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders