Improving Non-autoregressive Generation with Mixup Training
arXiv:2110.11115
Abstract
While pre-trained language models have achieved great success on various natural language understanding tasks, how to effectively leverage them into non-autoregressive generation tasks remains a challenge. To solve this problem, we present a non-autoregressive generation model based on pre-trained transformer models. To bridge the gap between autoregressive and non-autoregressive models, we propose a simple and effective iterative training method called MIx Source and pseudo Target (MIST). Unlike other iterative decoding methods, which sacrifice the inference speed to achieve better performance based on multiple decoding iterations, MIST works in the training stage and has no effect on inference time. Our experiments on three generation benchmarks including question generation, summarization and paraphrase generation, show that the proposed framework achieves the new state-of-the-art results for fully non-autoregressive models. We also demonstrate that our method can be used to a variety of pre-trained models. For instance, MIST based on the small pre-trained model also obtains comparable performance with seq2seq models.
References in corpus (14)
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- Non-Autoregressive Neural Machine Translation
- A Deep Generative Framework for Paraphrase Generation
- ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
- Neural Question Generation from Text: A Preliminary Study
- Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
- Imitation Learning for Non-Autoregressive Neural Machine Translation
- BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
- Glancing Transformer for Non-Autoregressive Neural Machine Translation
- Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
- Semi-Autoregressive Neural Machine Translation
- Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
- GLGE: A New General Language Generation Evaluation Benchmark