Pre-trained Language Model Representations for Language Generation
arXiv:1903.09722
Abstract
Pre-trained language model representations have been successful in a wide range of language understanding tasks. In this paper, we examine different strategies to integrate pre-trained representations into sequence to sequence models and apply it to neural machine translation and abstractive summarization. We find that pre-trained representations are most effective when added to the encoder network which slows inference by only 14%. Our experiments in machine translation show gains of up to 5.3 BLEU in a simulated resource-poor setup. While returns diminish with more labeled data, we still observe improvements when millions of sentence-pairs are available. Finally, on abstractive summarization we achieve a new state of the art on the full text version of CNN/DailyMail.
NAACL 2019
References in corpus (9)
- Teaching Machines to Read and Comprehend
- SGDR: Stochastic Gradient Descent with Warm Restarts
- A Deep Reinforced Model for Abstractive Summarization
- On Using Monolingual Corpora in Neural Machine Translation
- Layer Normalization
- Semi-supervised Sequence Learning
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- Bottom-Up Abstractive Summarization
Cited by in corpus (17)
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- Multilingual Denoising Pre-training for Neural Machine Translation
- vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
- Reducing Transformer Depth on Demand with Structured Dropout
- A Survey of Deep Learning Techniques for Neural Machine Translation
- ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
- Addressing Some Limitations of Transformers with Feedback Memory
- Encoder-Agnostic Adaptation for Conditional Language Generation
- On Learning Universal Representations Across Languages
- Neural Machine Translation: A Review of Methods, Resources, and Tools
- Diverse Pretrained Context Encodings Improve Document Translation
- Multimodal Conditionality for Natural Language Generation
- Improving AMR Parsing with Sequence-to-Sequence Pre-training
- Denoising based Sequence-to-Sequence Pre-training for Text Generation
- Discovering Useful Sentence Representations from Large Pretrained Language Models
- Unsupervised Pre-training for Natural Language Generation: A Literature Review
- Improving speech recognition models with small samples for air traffic control systems