Non-Autoregressive Machine Translation with Disentangled Context Transformer

Abstract

State-of-the-art neural machine translation models generate a translation from left to right and every step is conditioned on the previously generated tokens. The sequential nature of this generation process causes fundamental latency in inference since we cannot generate multiple tokens in each sentence in parallel. We propose an attention-masking based model, called Disentangled Context (DisCo) transformer, that simultaneously generates all tokens given different contexts. The DisCo transformer is trained to predict every output token given an arbitrary subset of the other reference tokens. We also develop the parallel easy-first inference algorithm, which iteratively refines every token in parallel and reduces the number of required iterations. Our extensive experiments on 7 translation directions with varying data sizes demonstrate that our model achieves competitive, if not better, performance compared to the state of the art in non-autoregressive machine translation while significantly reducing decoding time on average. Our code is available at https://github.com/facebookresearch/DisCo.

ICML 2020

References in corpus (13)

• Convolutional Sequence to Sequence Learning
• Mixed Precision Training
• Non-Autoregressive Neural Machine Translation
• Adaptive Computation Time for Recurrent Neural Networks
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Levenshtein Transformer
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• KERMIT: Generative Insertion-Based Modeling for Sequences
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Insertion-based Decoding with automatically Inferred Generation Order
• LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention

Cited by in corpus (15)

• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• Transformer-based Conditional Variational Autoencoder for Controllable Story Generation
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• Accelerating Transformer Inference for Translation via Parallel Decoding
• Glancing Transformer for Non-Autoregressive Neural Machine Translation
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• Progressive Generation of Long Text with Pretrained Language Models
• Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints
• Sequence-Level Training for Non-Autoregressive Neural Machine Translation
• Mask-Align: Self-Supervised Neural Word Alignment
• O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning
• Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsing
• Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers