Insertion-based Decoding with automatically Inferred Generation Order

Abstract

Conventional neural autoregressive decoding commonly assumes a fixed left-to-right generation order, which may be sub-optimal. In this work, we propose a novel decoding algorithm -- InDIGO -- which supports flexible sequence generation in arbitrary orders through insertion operations. We extend Transformer, a state-of-the-art sequence generation model, to efficiently implement the proposed approach, enabling it to be trained with either a pre-defined generation order or adaptive orders obtained from beam-search. Experiments on four real-world tasks, including word order recovery, machine translation, image caption and code generation, demonstrate that our algorithm can generate sequences following arbitrary orders, while achieving competitive or even better performance compared to the conventional left-to-right generation. The generated sequences show that InDIGO adopts adaptive generation orders based on input information.

arXiv commentCamera ready. Accepted by TACL

References in corpus (8)

• Sequence to Sequence Learning with Neural Networks
• Attention-Based Models for Speech Recognition
• A Neural Conversational Model
• Non-Autoregressive Neural Machine Translation
• Learning Latent Permutations with Gumbel-Sinkhorn Networks
• Latent Predictor Networks for Code Generation
• Non-Monotonic Sequential Text Generation
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model

Cited by in corpus (17)

• Defending Against Neural Fake News
• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• Semi-Autoregressive Training Improves Mask-Predict Decoding
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Real or Fake? Learning to Discriminate Machine from Human Generated Text
• Imputer: Sequence Modelling via Imputation and Dynamic Programming
• Neural Language Generation: Formulation, Methods, and Evaluation
• Strategies for Structuring Story Generation
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• Enabling Language Models to Fill in the Blanks
• Insertion-Deletion Transformer
• Neural Machine Translation: A Review and Survey
• An Empirical Study of Generation Order for Machine Translation
• Sequential Graph Dependency Parser
• Big Bidirectional Insertion Representations for Documents
• Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels