Non-autoregressive Transformer by Position Learning

Abstract

Non-autoregressive models are promising on various text generation tasks. Previous work hardly considers to explicitly model the positions of generated words. However, position modeling is an essential problem in non-autoregressive text generation. In this study, we propose PNAT, which incorporates positions as a latent variable into the text generative process. Experimental results show that PNAT achieves top results on machine translation and paraphrase generation tasks, outperforming several strong baselines.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Fast Decoding in Sequence Models using Discrete Latent Variables
• A Deep Generative Framework for Paraphrase Generation
• Imitation Learning for Non-Autoregressive Neural Machine Translation

Cited by in corpus (12)

• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting
• Glancing Transformer for Non-Autoregressive Neural Machine Translation
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Non-Autoregressive Neural Dialogue Generation
• Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
• LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention
• Non-Autoregressive Translation by Learning Target Categorical Codes
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation
• Modeling Coverage for Non-Autoregressive Neural Machine Translation
• NAST: A Non-Autoregressive Generator with Word Alignment for Unsupervised Text Style Transfer