Glancing Transformer for Non-Autoregressive Neural Machine Translation
arXiv:2008.07905
Abstract
Recent work on non-autoregressive neural machine translation (NAT) aims at improving the efficiency by parallel decoding without sacrificing the quality. However, existing NAT methods are either inferior to Transformer or require multiple decoding passes, leading to reduced speedup. We propose the Glancing Language Model (GLM), a method to learn word interdependency for single-pass parallel generation models. With GLM, we develop Glancing Transformer (GLAT) for machine translation. With only single-pass parallel decoding, GLAT is able to generate high-quality translation with 8-15 times speedup. Experiments on multiple WMT language directions show that GLAT outperforms all previous single pass non-autoregressive methods, and is nearly comparable to Transformer, reducing the gap to 0.25-0.9 BLEU points.
arXiv comment9 pages, 7 figures, ACL2021
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Non-Autoregressive Neural Machine Translation
- Aligned Cross Entropy for Non-Autoregressive Machine Translation
- Non-Autoregressive Machine Translation with Disentangled Context Transformer
- Semi-Autoregressive Training Improves Mask-Predict Decoding
- Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
- Imitation Learning for Non-Autoregressive Neural Machine Translation
- Non-autoregressive Transformer by Position Learning
- An EM Approach to Non-autoregressive Conditional Sequence Generation
- Cascaded Text Generation with Markov Transformers
- Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
Cited by in corpus (5)
- NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting
- Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
- Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
- Improving Non-autoregressive Generation with Mixup Training
- Non-Autoregressive Translation by Learning Target Categorical Codes