LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention

Abstract

Non-autoregressive translation (NAT) models generate multiple tokens in one forward pass and is highly efficient at inference stage compared with autoregressive translation (AT) methods. However, NAT models often suffer from the multimodality problem, i.e., generating duplicated tokens or missing tokens. In this paper, we propose two novel methods to address this issue, the Look-Around (LA) strategy and the Vocabulary Attention (VA) mechanism. The Look-Around strategy predicts the neighbor tokens in order to predict the current token, and the Vocabulary Attention models long-term token dependencies inside the decoder by attending the whole vocabulary for each position to acquire knowledge of which token is about to generate. %We also propose a dynamic bidirectional decoding approach to accelerate the inference process of the LAVA model while preserving the high-quality of the generated output. Our proposed model uses significantly less time during inference compared with autoregressive models and most other NAT models. Our experiments on four benchmarks (WMT14 En$\to$De, WMT14 De$\to$En, WMT16 Ro$\to$En and IWSLT14 De$\to$En) show that the proposed model achieves competitive performance compared with the state-of-the-art non-autoregressive and autoregressive models while significantly reducing the time cost in inference phase.

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Convolutional Sequence to Sequence Learning
• Neural Machine Translation in Linear Time
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• An Actor-Critic Algorithm for Sequence Prediction
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• Non-autoregressive Transformer by Position Learning
• Hint-Based Training for Non-Autoregressive Machine Translation

Cited by in corpus (4)

• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation