Cascaded Text Generation with Markov Transformers
arXiv:2006.01112
Abstract
The two dominant approaches to neural text generation are fully autoregressive models, using serial beam search decoding, and non-autoregressive models, using parallel decoding with no output dependencies. This work proposes an autoregressive model with sub-linear parallel time generation. Noting that conditional random fields with bounded context can be decoded in parallel, we propose an efficient cascaded decoding approach for generating high-quality output. To parameterize this cascade, we introduce a Markov transformer, a variant of the popular fully autoregressive model that allows us to simultaneously decode with specific autoregressive context cutoffs. This approach requires only a small modification from standard autoregressive training, while showing competitive accuracy/speed tradeoff compared to existing methods on five machine translation datasets.
arXiv commentNeurIPS 2020
References in corpus (29)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Auto-Encoding Variational Bayes
- Neural Machine Translation by Jointly Learning to Align and Translate
- Distilling the Knowledge in a Neural Network
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Language Models are Few-Shot Learners
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- Non-Autoregressive Neural Machine Translation
- Release Strategies and the Social Impacts of Language Models
- Neural Text Generation with Unlikelihood Training
- Neural Variational Inference and Learning in Belief Networks
- Levenshtein Transformer
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- Insertion Transformer: Flexible Sequence Generation via Insertion Operations
- Sequence-Level Knowledge Distillation
- Structured Prediction Cascades
- Challenges in Data-to-Document Generation
- Semi-Autoregressive Training Improves Mask-Predict Decoding
- A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
- Understanding Knowledge Distillation in Non-autoregressive Machine Translation
- Imitation Learning for Non-Autoregressive Neural Machine Translation
- Real or Fake? Learning to Discriminate Machine from Human Generated Text
- FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
- Semi-Autoregressive Neural Machine Translation
- POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
- Improving Non-autoregressive Neural Machine Translation with Monolingual Data
- End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification
- Speeding Up Neural Machine Translation Decoding by Cube Pruning