A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models

Abstract

Undirected neural sequence models such as BERT (Devlin et al., 2019) have received renewed interest due to their success on discriminative natural language understanding tasks such as question-answering and natural language inference. The problem of generating sequences directly from these models has received relatively little attention, in part because generating from undirected models departs significantly from conventional monotonic generation in directed sequence models. We investigate this problem by proposing a generalized model of sequence generation that unifies decoding in directed and undirected models. The proposed framework models the process of generation rather than the resulting sequence, and under this framework, we derive various neural sequence models as special cases, such as autoregressive, semi-autoregressive, and refinement-based non-autoregressive models. This unification enables us to adapt decoding algorithms originally developed for directed sequence models to undirected sequence models. We demonstrate this by evaluating various handcrafted and learned decoding strategies on a BERT-like machine translation model (Lample & Conneau, 2019). The proposed approach achieves constant-time translation results on par with linear-time translation results from the same undirected sequence model, while both are competitive with the state-of-the-art on WMT'14 English-German translation.

References in corpus (10)

• Cross-lingual Language Model Pretraining
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Non-Autoregressive Neural Machine Translation
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Insertion-based Decoding with automatically Inferred Generation Order
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• Semi-Autoregressive Neural Machine Translation
• FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow
• Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning

Cited by in corpus (14)

• Non-Autoregressive Machine Translation with Disentangled Context Transformer
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
• Cascaded Text Generation with Markov Transformers
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
• Logical Natural Language Generation from Open-Domain Tables
• XL-Editor: Post-editing Sentences with XLNet
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
• Fill in the Blanks: Imputing Missing Sentences for Larger-Context Neural Machine Translation
• Progressive Generation of Long Text with Pretrained Language Models
• PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation
• Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
• Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation