Language as a Latent Variable: Discrete Generative Models for Sentence Compression

Abstract

In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.

EMNLP 2016

References in corpus (1)

• DRAW: A Recurrent Neural Network For Image Generation

Cited by in corpus (11)

• Topic-Guided Variational Autoencoders for Text Generation
• Deep Recurrent Generative Decoder for Abstractive Text Summarization
• The Neural Noisy Channel
• A Multi-task Learning Approach for Improving Product Title Compression with User Search Log Data
• Unsupervised Dialog Structure Learning
• Learning Autocomplete Systems as a Communication Game
• Self-organized Hierarchical Softmax
• Discriminative Pre-training for Low Resource Title Compression in Conversational Grocery
• A novel repetition normalized adversarial reward for headline generation
• Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder
• Semi-supervised Autoencoding Projective Dependency Parsing