BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

Abstract

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

arXiv commentNeuralGen 2019; https://colab.research.google.com/drive/1MxKZGtQ9SSBjTK5ArsZ5LKhkztzg52RV

References in corpus (5)

• Cross-lingual Language Model Pretraining
• Pointer Sentinel Mixture Models
• Passage Re-ranking with BERT
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Assessing BERT's Syntactic Abilities

Cited by in corpus (29)

• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Learning Video Representations using Contrastive Bidirectional Transformer
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Structured Denoising Diffusion Models in Discrete State-Spaces
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• FrameAxis: Characterizing Microframe Bias and Intensity with Word Embedding
• A Generalized Framework of Sequence Generation with Application to Undirected Sequence Models
• Self-Training Vision Language BERTs with a Unified Conditional Model
• Distilling Knowledge Learned in BERT for Text Generation
• Energy-based View of Retrosynthesis
• Enabling Language Models to Fill in the Blanks
• CBAG: Conditional Biomedical Abstract Generation
• Deleter: Leveraging BERT to Perform Unsupervised Successive Text Compression
• L2RS: A Learning-to-Rescore Mechanism for Automatic Speech Recognition
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
• XL-Editor: Post-editing Sentences with XLNet
• BERT as a Teacher: Contextual Embeddings for Sequence-Level Reward
• Combining Neural Language Models for WordSense Induction
• On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
• Energy-Based Models with Applications to Speech and Language Processing
• Rewriting Meaningful Sentences via Conditional BERT Sampling and an application on fooling text classifiers
• On Language Models for Creoles
• Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task
• How Good is Artificial Intelligence at Automatically Answering Consumer Questions Related to Alzheimer's Disease?
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation
• Pre-Training Transformers as Energy-Based Cloze Models
• Beyond In-Place Corruption: Insertion and Deletion In Denoising Probabilistic Models