Consistency Regularization for Cross-Lingual Fine-Tuning
arXiv:2106.08226
Abstract
Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks, including text classification, question answering, and sequence labeling.
ACL-2021
References in corpus (7)
- Exploiting Similarities among Languages for Machine Translation
- Learning Discrete Representations via Information Maximizing Self-Augmented Training
- FreeLB: Enhanced Adversarial Training for Natural Language Understanding
- There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
- Adversarial Training for Large Neural Language Models
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
- XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering