Aligned Cross Entropy for Non-Autoregressive Machine Translation
arXiv:2004.01655
Abstract
Non-autoregressive machine translation models significantly speed up decoding by allowing for parallel prediction of the entire target sequence. However, modeling word order is more challenging due to the lack of autoregressive factors in the model. This difficultly is compounded during training with cross entropy loss, which can highly penalize small shifts in word order. In this paper, we propose aligned cross entropy (AXE) as an alternative loss function for training of non-autoregressive models. AXE uses a differentiable dynamic program to assign loss based on the best possible monotonic alignment between target tokens and model predictions. AXE-based training of conditional masked language models (CMLMs) substantially improves performance on major WMT benchmarks, while setting a new state of the art for non-autoregressive models.
References in corpus (10)
- Pay Less Attention with Lightweight and Dynamic Convolutions
- Levenshtein Transformer
- Insertion Transformer: Flexible Sequence Generation via Insertion Operations
- Non-Autoregressive Machine Translation with Disentangled Context Transformer
- Semi-Autoregressive Training Improves Mask-Predict Decoding
- Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
- Understanding Knowledge Distillation in Non-autoregressive Machine Translation
- Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
- FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
- Hint-Based Training for Non-Autoregressive Machine Translation
Cited by in corpus (13)
- Neural Program Repair with Execution-based Backpropagation
- Non-Autoregressive Machine Translation with Disentangled Context Transformer
- Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
- FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition
- Accelerating Transformer Inference for Translation via Parallel Decoding
- Glancing Transformer for Non-Autoregressive Neural Machine Translation
- Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
- ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
- Fast Interleaved Bidirectional Sequence Generation
- Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation
- Sequence-Level Training for Non-Autoregressive Neural Machine Translation
- O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
- Learning Energy-Based Approximate Inference Networks for Structured Applications in NLP