Training Multilingual Pre-trained Language Model with Byte-level Subwords

Abstract

The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. One of the fundamental components in pre-trained language models is the vocabulary, especially for training multilingual models on many different languages. In the technical report, we present our practices on training multilingual pre-trained language models with BBPE: Byte-Level BPE (i.e., Byte Pair Encoding). In the experiment, we adopted the architecture of NEZHA as the underlying pre-trained language model and the results show that NEZHA trained with byte-level subwords consistently outperforms Google multilingual BERT and vanilla NEZHA by a notable margin in several multilingual NLU tasks. We release the source code of our byte-level vocabulary building tools and the multilingual pre-trained language models.

References in corpus (9)

• Language Models are Few-Shot Learners
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Mixed Precision Training
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• ERNIE: Enhanced Representation through Knowledge Integration
• Horovod: fast and easy distributed deep learning in TensorFlow
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• Neural Machine Translation with Byte-Level Subwords

Cited by in corpus (2)

• Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision
• Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languages