Towards Making the Most of BERT in Neural Machine Translation
arXiv:1908.05672
Abstract
GPT-2 and BERT demonstrate the effectiveness of using pre-trained language models (LMs) on various natural language processing tasks. However, LM fine-tuning often suffers from catastrophic forgetting when applied to resource-rich tasks. In this work, we introduce a concerted training framework (CTNMT) that is the key to integrate the pre-trained LMs to neural machine translation (NMT). Our proposed CTNMT consists of three techniques: a) asymptotic distillation to ensure that the NMT model can retain the previous pre-trained knowledge; b) a dynamic switching gate to avoid catastrophic forgetting of pre-trained knowledge; and c) a strategy to adjust the learning paces according to a scheduled policy. Our experiments in machine translation show CTNMT gains of up to 3 BLEU score on the WMT14 English-German language pair which even surpasses the previous state-of-the-art pre-training aided NMT by 1.4 BLEU score. While for the large WMT14 English-French task with 40 millions of sentence-pairs, our base model still significantly improves upon the state-of-the-art Transformer big model by more than 1 BLEU score. The code and model can be downloaded from https://github.com/bytedance/neurst/ tree/master/examples/ctnmt.
arXiv comment10pages. the same as AAAI 2020 version, reformated with additional link to github repository
References in corpus (6)
- Distributed Representations of Words and Phrases and their Compositionality
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Natural Language Processing (almost) from Scratch
- Distributed Representations of Sentences and Documents
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Skip-Thought Vectors
Cited by in corpus (9)
- Multilingual Denoising Pre-training for Neural Machine Translation
- Incorporating BERT into Parallel Sequence Decoding with Adapters
- Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation
- Distilling Knowledge Learned in BERT for Text Generation
- Acquiring Knowledge from Pre-trained Model to Neural Machine Translation
- Code-switching pre-training for neural machine translation
- Unsupervised Pre-training for Natural Language Generation: A Literature Review
- Iterative Domain-Repaired Back-Translation
- Volctrans Parallel Corpus Filtering System for WMT 2020