BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

Abstract

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

NeuralGen 2019; https://colab.research.google.com/drive/1MxKZGtQ9SSBjTK5ArsZ5LKhkztzg52RV

References in corpus (4)

• Cross-lingual Language Model Pretraining
• Pointer Sentinel Mixture Models
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Assessing BERT's Syntactic Abilities

Cited by in corpus (12)

• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Enabling Language Models to Fill in the Blanks
• L2RS: A Learning-to-Rescore Mechanism for Automatic Speech Recognition
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• XL-Editor: Post-editing Sentences with XLNet
• Combining Neural Language Models for WordSense Induction
• Energy-Based Models with Applications to Speech and Language Processing
• On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
• On Language Models for Creoles
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation
• Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models
• Pre-Training Transformers as Energy-Based Cloze Models