Denoising based Sequence-to-Sequence Pre-training for Text Generation
arXiv:1908.08206
Abstract
This paper presents a new sequence-to-sequence (seq2seq) pre-training method PoDA (Pre-training of Denoising Autoencoders), which learns representations suitable for text generation tasks. Unlike encoder-only (e.g., BERT) or decoder-only (e.g., OpenAI GPT) pre-training approaches, PoDA jointly pre-trains both the encoder and decoder by denoising the noise-corrupted text, and it also has the advantage of keeping the network architecture unchanged in the subsequent fine-tuning stage. Meanwhile, we design a hybrid model of Transformer and pointer-generator networks as the backbone architecture for PoDA. We conduct experiments on two text generation tasks: abstractive summarization, and grammatical error correction. Results on four datasets show that PoDA can improve model performance over strong baselines without using any task-specific techniques and significantly speed up convergence.
arXiv commentAccepted to EMNLP 2019
References in corpus (17)
- Sequence to Sequence Learning with Neural Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Convolutional Sequence to Sequence Learning
- Cross-lingual Language Model Pretraining
- A Neural Conversational Model
- Building Natural-Language Generation Systems
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Semi-supervised Sequence Learning
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- A Hierarchical Neural Autoencoder for Paragraphs and Documents
- Faithful to the Original: Fact Aware Neural Abstractive Summarization
- Neural Language Correction with Character-Based Attention
- Get To The Point: Summarization with Pointer-Generator Networks
- Reaching Human-level Performance in Automatic Grammatical Error Correction: An Empirical Study
- Pre-trained Language Model Representations for Language Generation
- Global Encoding for Abstractive Summarization