Semi-Autoregressive Training Improves Mask-Predict Decoding
arXiv:2001.08785
Abstract
The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mask-predict, producing training examples that contain model predictions as part of their inputs. Models trained with SMART produce higher-quality translations when using mask-predict decoding, effectively closing the remaining performance gap with fully autoregressive models.
References in corpus (4)
Cited by in corpus (16)
- Aligned Cross Entropy for Non-Autoregressive Machine Translation
- Accelerating Transformer Inference for Translation via Parallel Decoding
- Neural Language Generation: Formulation, Methods, and Evaluation
- Glancing Transformer for Non-Autoregressive Neural Machine Translation
- Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
- Cascaded Text Generation with Markov Transformers
- Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
- Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
- Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict
- Improved Mask-CTC for Non-Autoregressive End-to-End ASR
- Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation
- Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
- Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
- Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain
- Towards Variable-Length Textual Adversarial Attacks
- Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey