Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Abstract

In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.

EMNLP 2016

References in corpus (7)

• Auto-Encoding Variational Bayes
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• DRAW: A Recurrent Neural Network For Image Generation
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Latent Predictor Networks for Code Generation

Cited by in corpus (31)

• Get To The Point: Summarization with Pointer-Generator Networks
• MeanSum: A Neural Model for Unsupervised Multi-document Abstractive Summarization
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• Topic-Guided Variational Autoencoders for Text Generation
• Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols
• Actor-Critic based Training Framework for Abstractive Summarization
• Deep Recurrent Generative Decoder for Abstractive Text Summarization
• A Tutorial on Deep Latent Variable Models of Natural Language
• A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss
• The Neural Noisy Channel
• A Multi-task Learning Approach for Improving Product Title Compression with User Search Log Data
• Incorporating Discriminator in Sentence Generation: a Gibbs Sampling Method
• Adapting the Neural Encoder-Decoder Framework from Single to Multi-Document Summarization
• Unsupervised Dialog Structure Learning
• Multi-space Variational Encoder-Decoders for Semi-supervised Labeled Sequence Transduction
• Learning Autocomplete Systems as a Communication Game
• Abstractive Dialogue Summarization with Sentence-Gated Modeling Optimized by Dialogue Acts
• Latent Programmer: Discrete Latent Codes for Program Synthesis
• Extractive Summary as Discrete Latent Variables
• Closed-Book Training to Improve Summarization Encoder Memory
• Unsupervised Learning of Syntactic Structure with Invertible Neural Projections
• Simple Unsupervised Summarization by Contextual Matching
• Weakly-supervised Neural Semantic Parsing with a Generative Ranker
• Self-organized Hierarchical Softmax
• Multi-Source Pointer Network for Product Title Summarization
• Dependency Grammar Induction with a Neural Variational Transition-based Parser
• Discriminative Pre-training for Low Resource Title Compression in Conversational Grocery
• Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder
• Semi-supervised Autoencoding Projective Dependency Parsing
• A novel repetition normalized adversarial reward for headline generation
• Cross Copy Network for Dialogue Generation