KERMIT: Generative Insertion-Based Modeling for Sequences

Abstract

We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x,y)$ to learn the joint distribution $p(x,y)$, and optionally mix in unpaired data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we have access to the conditionals $p(x\mid y)$ and $p(y\mid x)$ in both directions. We can also sample from the joint distribution or the marginals. The model supports both serial fully autoregressive decoding and parallel partially autoregressive decoding, with the latter exhibiting an empirically logarithmic runtime. We demonstrate through experiments in machine translation, representation learning, and zero-shot cloze question answering that our unified approach is capable of matching or exceeding the performance of dedicated state-of-the-art systems across a wide range of tasks without the need for problem-specific architectural adaptation.

William Chan, Nikita Kitaev, Kelvin Guu, and Mitchell Stern contributed equally

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• ERNIE: Enhanced Representation through Knowledge Integration
• Effective Approaches to Attention-based Neural Machine Translation
• Transforming Question Answering Datasets Into Natural Language Inference Datasets
• Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement

Cited by in corpus (28)

• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• WaveGrad: Estimating Gradients for Waveform Generation
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• Neural Language Generation: Formulation, Methods, and Evaluation
• Imputer: Sequence Modelling via Imputation and Dynamic Programming
• Style Example-Guided Text Generation using Generative Adversarial Transformers
• Duplex Sequence-to-Sequence Learning for Reversible Machine Translation
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade
• XL-Editor: Post-editing Sentences with XLNet
• Insertion-Deletion Transformer
• Which *BERT? A Survey Organizing Contextualized Encoders
• Generating Music with a Self-Correcting Non-Chronological Autoregressive Model
• Contextual Lensing of Universal Sentence Representations
• Progressive Generation of Long Text with Pretrained Language Models
• Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation
• Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring
• Syntax-driven Iterative Expansion Language Models for Controllable Text Generation
• Computer Assisted Translation with Neural Quality Estimation and Automatic Post-Editing
• Discovering Useful Sentence Representations from Large Pretrained Language Models
• Insertion-Based Modeling for End-to-End Automatic Speech Recognition
• Cue-word Driven Neural Response Generation with a Shrinking Vocabulary
• Teaching Machine Comprehension with Compositional Explanations
• Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation
• Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels
• Big Bidirectional Insertion Representations for Documents
• Toward Streaming ASR with Non-Autoregressive Insertion-based Model