Insertion Transformer: Flexible Sequence Generation via Insertion Operations

Abstract

We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be inserted anywhere in the sequence during decoding. This flexibility confers a number of advantages: for instance, not only can our model be trained to follow specific orderings such as left-to-right generation or a binary tree traversal, but it can also be trained to maximize entropy over all valid insertions for robustness. In addition, our model seamlessly accommodates both fully autoregressive generation (one insertion at a time) and partially autoregressive generation (simultaneous insertions at multiple locations). We validate our approach by analyzing its performance on the WMT 2014 English-German machine translation task under various settings for training and decoding. We find that the Insertion Transformer outperforms many prior non-autoregressive approaches to translation at comparable or better levels of parallelism, and successfully recovers the performance of the original Transformer while requiring only logarithmically many iterations during decoding.

Cited by in corpus (69)

• Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition
• Defending Against Neural Fake News
• Aligned Cross Entropy for Non-Autoregressive Machine Translation
• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano Compositions
• Semi-Autoregressive Training Improves Mask-Predict Decoding
• Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• WaveGrad: Estimating Gradients for Waveform Generation
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• Understanding Knowledge Distillation in Non-autoregressive Machine Translation
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition
• BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
• Imputer: Sequence Modelling via Imputation and Dynamic Programming
• Neural Language Generation: Formulation, Methods, and Evaluation
• Imitation Attacks and Defenses for Black-box Machine Translation Systems
• Strategies for Structuring Story Generation
• Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information
• Style Example-Guided Text Generation using Generative Adversarial Transformers
• Learning Structural Edits via Incremental Tree Transformations
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• Cascaded Text Generation with Markov Transformers
• Lexically Constrained Neural Machine Translation with Levenshtein Transformer
• Neural Machine Translation: Challenges, Progress and Future
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
• Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• Insertion-Deletion Transformer
• Improved Mask-CTC for Non-Autoregressive End-to-End ASR
• Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict
• Contextual Lensing of Universal Sentence Representations
• Exploring Dynamic Selection of Branch Expansion Orders for Code Generation
• Which *BERT? A Survey Organizing Contextualized Encoders
• Nutribullets Hybrid: Multi-document Health Summarization
• Generating Music with a Self-Correcting Non-Chronological Autoregressive Model
• Progressive Generation of Long Text with Pretrained Language Models
• Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation
• Fast Interleaved Bidirectional Sequence Generation
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Faster Re-translation Using Non-Autoregressive Model For Simultaneous Neural Machine Translation
• Syntax-driven Iterative Expansion Language Models for Controllable Text Generation
• Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain
• Insertion-Based Modeling for End-to-End Automatic Speech Recognition
• Orthros: Non-autoregressive End-to-end Speech Translation with Dual-decoder
• Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
• Computer Assisted Translation with Neural Quality Estimation and Automatic Post-Editing
• Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation
• Learning Non-Monotonic Automatic Post-Editing of Translations from Human Orderings
• Cue-word Driven Neural Response Generation with a Shrinking Vocabulary
• Neural Machine Translation: A Review and Survey
• Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer
• Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
• An Empirical Study of Generation Order for Machine Translation
• IGA : An Intent-Guided Authoring Assistant
• Don't Parse, Insert: Multilingual Semantic Parsing with Insertion Based Decoding
• Recursive Top-Down Production for Sentence Generation with Latent Trees
• Big Bidirectional Insertion Representations for Documents
• Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels
• Sentence-Permuted Paragraph Generation
• Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systems
• Sequential Graph Dependency Parser
• EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints
• Incorporating a Local Translation Mechanism into Non-autoregressive Translation
• Toward Streaming ASR with Non-Autoregressive Insertion-based Model
• Data-to-text Generation by Splicing Together Nearest Neighbors