Exploring Benefits of Transfer Learning in Neural Machine Translation
arXiv:2001.01622
Abstract
Neural machine translation is known to require large numbers of parallel training sentences, which generally prevent it from excelling on low-resource language pairs. This thesis explores the use of cross-lingual transfer learning on neural networks as a way of solving the problem with the lack of resources. We propose several transfer learning approaches to reuse a model pretrained on a high-resource language pair. We pay particular attention to the simplicity of the techniques. We study two scenarios: (a) when we reuse the high-resource model without any prior modifications to its training process and (b) when we can prepare the first-stage high-resource model for transfer learning in advance. For the former scenario, we present a proof-of-concept method by reusing a model trained by other researchers. In the latter scenario, we present a method which reaches even larger improvements in translation performance. Apart from proposed techniques, we focus on an in-depth analysis of transfer learning techniques and try to shed some light on transfer learning improvements. We show how our techniques address specific problems of low-resource languages and are suitable even in high-resource transfer learning. We evaluate the potential drawbacks and behavior by studying transfer learning in various situations, for example, under artificially damaged training corpora, or with fixed various model parts.
arXiv commentDefended PhD thesis
References in corpus (17)
- Efficient Estimation of Word Representations in Vector Space
- Distilling the Knowledge in a Neural Network
- Sequence to Sequence Learning with Neural Networks
- Overcoming catastrophic forgetting in neural networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Neural Architecture Search with Reinforcement Learning
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Training Tips for the Transformer Model
- Character-based Neural Machine Translation
- Measuring Catastrophic Forgetting in Neural Networks
- Fast Domain Adaptation for Neural Machine Translation
- Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- On weight initialization in deep neural networks
- Measuring the Effects of Data Parallelism on Neural Network Training
- Improving Statistical Machine Translation for a Resource-Poor Language Using Related Resource-Rich Languages
- Domain specialization: a post-training domain adaptation for Neural Machine Translation
- A Comparative Study of Word Embeddings for Reading Comprehension