Unsupervised Pretraining for Sequence to Sequence Learning

Abstract

This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that pretraining improves the generalization of seq2seq models. We achieve state-of-the art results on the WMT English$\to$German task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves a significant improvement of 1.3 BLEU from the previous best models on both WMT'14 and WMT'15 English$\to$German. We also conduct human evaluations on abstractive summarization and find that our method outperforms a purely supervised learning baseline in a statistically significant manner.

Updated to accepted EMNLP 2017 version

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Recurrent Neural Network Regularization
• On Using Monolingual Corpora in Neural Machine Translation
• Achieving Human Parity in Conversational Speech Recognition
• Grammar as a Foreign Language
• Learning to Generate Reviews and Discovering Sentiment
• Zero-Resource Translation with Multi-Lingual Neural Machine Translation
• Unsupervised Learning of Predictors from Unpaired Input-Output Samples

Cited by in corpus (9)

• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• Learning Longer-term Dependencies in RNNs with Auxiliary Losses
• Large Scale Language Modeling: Converging on 40GB of Text in Four Hours
• Object category learning and retrieval with weak supervision
• Chinese User Service Intention Classification Based on Hybrid Neural Network
• Learning Contextualized Sentence Representations for Document-Level Neural Machine Translation
• Language Transfer of Audio Word2Vec: Learning Audio Segment Representations without Target Language Data
• Joint Text and Label Generation for Spoken Language Understanding