Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models

Abstract

Denoising diffusion probabilistic models (DDPMs) have shown impressive results on sequence generation by iteratively corrupting each example and then learning to map corrupted versions back to the original. However, previous work has largely focused on in-place corruption, adding noise to each pixel or token individually while keeping their locations the same. In this work, we consider a broader class of corruption processes and denoising models over sequence data that can insert and delete elements, while still being efficient to train and sample from. We demonstrate that these models outperform standard in-place models on an arithmetic sequence task, and that when trained on the text8 dataset they can be used to fix spelling errors without any fine-tuning.

arXiv commentAccepted at the ICML 2021 Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models (poster)

References in corpus (12)

• Denoising Diffusion Probabilistic Models
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Structured Denoising Diffusion Models in Discrete State-Spaces
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• Task Loss Estimation for Sequence Prediction
• Neural Networks for Modeling Source Code Edits
• Learning Structural Edits via Incremental Tree Transformations
• Insertion-Deletion Transformer
• Optimal Completion Distillation for Sequence Learning
• Discrete Object Generation with Reversible Inductive Construction