BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining

Abstract

In this paper, we propose BANG, a new pretraining model to Bridge the gap between Autoregressive (AR) and Non-autoregressive (NAR) Generation. AR and NAR generation can be uniformly regarded as to what extent previous tokens can be attended, and BANG bridges AR and NAR generation by designing a novel model structure for large-scale pretraining. The pretrained BANG model can simultaneously support AR, NAR and semi-NAR generation to meet different requirements. Experiments on question generation (SQuAD 1.1), summarization (XSum) and dialogue generation (PersonaChat) show that BANG improves NAR and semi-NAR performance significantly as well as attaining comparable performance with strong AR pretrained models. Compared with the semi-NAR strong baselines, BANG achieves absolute improvements of 14.01 and 5.24 in the overall scores of SQuAD 1.1 and XSum, respectively. In addition, BANG achieves absolute improvements of 10.73, 6.39 and 5.90 in the overall scores of SQuAD, XSUM and PersonaChat respectively compared with the strong NAR baselines.

arXiv commentAccepted by ICML 2021

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Non-Autoregressive Neural Machine Translation
• Levenshtein Transformer
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Sequence-Level Knowledge Distillation
• DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• Non-Autoregressive Text Generation with Pre-trained Language Models
• GLGE: A New General Language Generation Evaluation Benchmark

Cited by in corpus (4)

• Paradigm Shift in Natural Language Processing
• A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition
• Improving Non-autoregressive Generation with Mixup Training
• EL-Attention: Memory Efficient Lossless Attention for Generation