Insertion-Deletion Transformer
arXiv:2001.05540
Abstract
We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation. The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase. The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution of deletions over the current output hypothesis. The training method is a principled and simple algorithm, where the deletion model obtains its signal directly on-policy from the insertion model output. We demonstrate the effectiveness of our Insertion-Deletion Transformer on synthetic translation tasks, obtaining significant BLEU score improvement over an insertion-only model.
Accepted as an Extended Abstract at the Workshop of Neural Generation and Translation (WNGT 2019) at EMNLP 2019
References in corpus (6)
- Sequence to Sequence Learning with Neural Networks
- Levenshtein Transformer
- Insertion Transformer: Flexible Sequence Generation via Insertion Operations
- KERMIT: Generative Insertion-Based Modeling for Sequences
- Non-Monotonic Sequential Text Generation
- Insertion-based Decoding with automatically Inferred Generation Order
Cited by in corpus (5)
- Structured Denoising Diffusion Models in Discrete State-Spaces
- WaveGrad: Estimating Gradients for Waveform Generation
- Neural Language Generation: Formulation, Methods, and Evaluation
- Learning Non-Monotonic Automatic Post-Editing of Translations from Human Orderings
- Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models