Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
arXiv:1909.11861
Abstract
In this paper, we investigate the problem of training neural machine translation (NMT) systems with a dataset of more than 40 billion bilingual sentence pairs, which is larger than the largest dataset to date by orders of magnitude. Unprecedented challenges emerge in this situation compared to previous NMT work, including severe noise in the data and prohibitively long training time. We propose practical solutions to handle these issues and demonstrate that large-scale pretraining significantly improves NMT performance. We are able to push the BLEU score of WMT17 Chinese-English dataset to 32.3, with a significant performance boost of +3.2 over existing state-of-the-art results.
References in corpus (25)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Cross-lingual Language Model Pretraining
- One weird trick for parallelizing convolutional neural networks
- Sequence Level Training with Recurrent Neural Networks
- Don't Decay the Learning Rate, Increase the Batch Size
- Achieving Human Parity on Automatic Chinese to English News Translation
- Semi-supervised Sequence Learning
- Synthetic and Natural Noise Both Break Neural Machine Translation
- A Simple, Fast Diverse Decoding Algorithm for Neural Generation
- Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
- Learning Factored Representations in a Deep Mixture of Experts
- Understanding Back-Translation at Scale
- Massive Exploration of Neural Machine Translation Architectures
- Massively Multilingual Neural Machine Translation
- Zero-Resource Translation with Multi-Lingual Neural Machine Translation
- Variational Neural Machine Translation
- A Tutorial on Deep Latent Variable Models of Natural Language
- Modeling Source Syntax for Neural Machine Translation
- Is Word Segmentation Necessary for Deep Learning of Chinese Representations?
- Sequence to Sequence Mixture Model for Diverse Machine Translation
- Hard but Robust, Easy but Sensitive: How Encoder and Decoder Perform in Neural Machine Translation
- Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints
- A Stochastic Decoder for Neural Machine Translation